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Highlights 

Highlights  of  GAO-12-352,  a  report  to  the 
Ranking  Member,  Subcommittee  on  Water 
Resources  and  Environment,  Committee  on 
Transportation  and  Infrastructure,  House  of 
Representatives 

Why  GAO  Did  This  Study 

Section  2034  of  the  Water  Resources 
Development  Act  of  2007  requires  that 
certain  U.S.  Army  Corps  of  Engineers 
(Corps)  civil  works  project  studies 
undergo  independent  external  peer 
review  to  assess  the  adequacy  and 
acceptability  of  the  methods,  models, 
and  analyses  used.  In  the  act, 

Congress  established  a  7-year  trial 
period  for  this  requirement  and  also 
required  the  Corps  to  submit  two 
reports  on  its  experiences  with  the 
peer  review  process. 

GAO  was  asked  to  examine  (1)  the 
number  of  Corps  project  studies  that 
have  undergone  independent  peer 
review  in  response  to  section  2034, 

(2)  the  cost  of  these  peer  reviews, 

(3)  the  extent  to  which  the  Corps’ 
process  for  determining  if  a  project 
study  is  subject  to  peer  review  is 
consistent  with  section  2034,  (4)  the 
process  the  Corps  uses  to  ensure  that 
the  contractors  it  hires  and  the  experts 
the  contractors  select  to  review  project 
studies  are  independent  and  free  from 
conflicts  of  interest,  and  (5)  the  extent 
to  which  peer  review  recommendations 
have  been  incorporated  into  project 
studies.  GAO  reviewed  relevant  laws, 
agency  guidance,  and  documents  and 
interviewed  Corps  officials  and 
contractors. 

What  GAO  Recommends 

GAO  recommends  that  the  Department 
of  Defense  direct  the  Corps  to,  among 
other  actions,  better  track  peer  review 
studies,  revise  the  criteria  for 
determining  which  studies  undergo 
peer  review  and  the  timing  of  these 
reviews,  and  improve  its  process  for 
ensuring  contractor  independence.  The 
department  generally  concurred  with 
these  recommendations. 

View  GAO-12-352.  For  more  information, 
contact  Anu  K.  Mittal  at  (202)  512-3841  or 
mittala@gao.gov. 
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What  GAO  Found 

Since  enactment  of  the  Water  Resources  Development  Act  of  2007,  49  project 
studies  have  undergone  peer  review  but  it  is  unclear  how  many  were  performed 
in  response  to  section  2034  requirements  because  the  Army  Corps  of  Engineers 
(Corps)  does  not  make  specific  determinations  or  track  if  a  peer  review  is  being 
conducted  under  section  2034.  In  February  2011,  in  response  to  section  2034, 
the  Corps  submitted  its  initial  report  to  Congress  summarizing  its  implementation 
of  the  peer  review  process.  In  its  report,  however,  the  Corps  did  not  distinguish 
which  studies  had  been  selected  for  peer  review  in  accordance  with  section  2034 
and  therefore,  did  not  provide  Congress  information  that  would  help  decision 
makers  evaluate  the  requirements  of  section  2034  at  the  end  of  the  trial  period. 

The  49  peer  reviews  resulted  in  both  direct  and  indirect  costs.  Specifically,  these 
peer  reviews  resulted  in  direct  costs  of  over  $9  million  in  contract  costs  and  fees. 
In  addition,  Corps  staff  resources  were  used  to  manage  the  reviews,  although 
these  costs  are  not  fully  quantifiable.  Furthermore,  the  addition  of  peer  review  to 
the  Corps  study  process  has  resulted  in  indirect  costs  by  altering  project  study 
schedules  to  allow  for  time  needed  to  complete  peer  reviews.  In  some  cases 
where  a  peer  review  was  not  planned  during  the  early  stages  of  the  study 
process,  significant  delays  to  project  studies  occurred  while  funds  were  sought  to 
pay  for  the  peer  review.  In  contrast,  according  to  some  Corps  officials,  when 
project  managers  have  built  in  time  and  identified  funding  for  peer  reviews  early, 
the  process  has  had  less  of  an  impact  on  project  study  schedules. 

The  Corps’  process  for  determining  whether  a  project  study  is  subject  to  peer 
review  is  more  expansive  than  section  2034  requirements  because  it  uses 
broader  criteria,  resulting  in  peer  reviews  of  studies  outside  the  scope  of  section 
2034.  In  addition,  the  process  the  Corps  uses  does  not  include  the  flexibility 
provided  in  section  2034,  which  allows  for  the  exclusion  of  certain  project  studies 
from  peer  review.  Moreover,  some  studies  are  undergoing  peer  reviews  that  do 
not  warrant  it,  according  to  some  Corps  officials  GAO  spoke  with. 

The  Corps  has  a  process  to  review  general  information  on  contractors’  conflicts 
of  interest  and  independence  when  selecting  them  to  establish  peer  review 
panels,  but  it  does  not  have  a  process  for  reviewing  project-level  information  on 
conflicts  of  interest  and  independence.  As  a  result,  it  cannot  be  assured  that 
contractors  do  not  have  conflicts  at  the  project-level.  In  contrast,  the  Corps’ 
contractors  do  have  a  process  for  reviewing  information  related  to  conflicts  of 
interest  and  the  independence  of  experts  selected  for  each  peer  review  panel. 

The  Corps  has  adopted  and  incorporated  into  its  project  study  reports  most  of  the 
peer  review  recommendations  it  has  received.  Doing  so  has  resulted  in  some 
technical  improvements  to  study  reports  but  generally  has  not  changed  the 
Corps’  decisions  about  project  alternatives,  in  part  because  the  peer  review 
process  occurs  too  late  in  the  project  study  process  to  affect  decision  making, 
according  to  some  Corps  officials  GAO  spoke  with.  As  a  result,  some 
recommendations  about  alternatives  may  not  have  been  implemented  because 
the  decision  on  the  preferred  design  had  already  been  made. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


March  8,  2012 

The  Honorable  Timothy  Bishop 
Ranking  Member 

Subcommittee  on  Water  Resources  and  Environment 
Committee  on  Transportation  and  Infrastructure 
House  of  Representatives 

Dear  Mr.  Bishop: 

Through  its  civil  works  program,  the  U.S.  Army  Corps  of  Engineers 
(Corps)  constructs,  operates,  and  maintains  thousands  of  civil  works 
projects  related  to  water  resources  across  the  United  States.1  These 
projects  aim  to  provide  safe  and  reliable  waterways;  reduce  risk  to 
people,  homes,  and  communities  from  flooding  and  coastal  storms; 
restore  and  protect  the  environment;  and  address  water  resources 
challenges.  A  Corps  civil  works  project  generally  starts  with  a  study  of  a 
water  resources  issue  and  the  development  of  various  alternatives  to 
address  it.  Such  studies  can  span  the  full  range  of  Corps  civil  works 
projects,  and  can  include  those  that  are  small  and  low  impact  and  others 
that  are  large  and  complex,  with  potentially  significant  economic  and 
environmental  impacts.  Through  its  civil  works  program,  the  Corps 
operates  50  centers  of  expertise  and  seven  research  laboratories  that 
assist  its  eight  divisions  and  38  district  offices  in  the  planning,  design,  and 
technical  review  of  civil  works  projects.2  Six  of  these  centers  are  focused 
on  the  quality  and  effectiveness  of  water  resources  planning  and  are 
referred  to  as  “planning  centers  of  expertise.” 

Through  its  civil  works  projects,  the  Corps  provides  vital  public 
engineering  services  in  peace  and  war  to  strengthen  the  nation’s  security, 


^he  Corps  has  both  a  military  and  a  civil  works  program.  The  military  program  provides, 
among  other  things,  engineering  and  construction  services  to  other  federal  agencies  and 
foreign  governments,  and  the  civil  works  program  is  responsible  for  investigating, 
developing,  and  maintaining  water  resources  projects.  This  report  discusses  only  the  civil 
works  program. 

2The  centers  of  expertise  are  designated  individuals  or  organizations — located  either  in 
district  offices,  division  offices,  or  research  laboratories — with  capability  or  expertise  in  a 
specialized  area.  The  Corps  designates  employees  at  various  levels  within  the  Corps  to 
oversee,  manage,  and  coordinate  the  centers. 
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energize  the  economy,  and  reduce  risks  from  disasters.  These  projects 
involve  navigation  and  flood  control  activities,  environmental  restoration, 
and  emergency  response — most  recently  including  emergency  response 
to  Missouri  River  flooding  and  rebuilding  after  Hurricane  Katrina. 

Technical  errors  in  past  studies  of  Corps  projects,  however,  had  raised 
concerns  about  the  effectiveness  of  the  Corps’  internal  review  processes 
and  the  quality  of  the  studies  that  the  Corps  used  as  a  basis  for  its  civil 
works  projects.3  For  example,  in  March  2006,  we  reported  that  certain 
studies  completed  by  the  Corps  from  1992  through  2002  were  fraught 
with  errors,  mistakes,  and  miscalculations  and  used  invalid  assumptions 
and  outdated  data.4  We  also  reported  that  these  Corps  studies 
understated  costs,  overstated  benefits,  and  did  not  provide  a  reasonable 
basis  for  decision  making.  Similar  findings  have  been  documented  in 
reviews  by  the  National  Academy  of  Sciences  (NAS)  and  other 
organizations,  which  concluded  that  the  Corps’  review  processes  needed 
to  be  strengthened. 

In  the  wake  of  these  reports,  Congress  passed  section  2034  of  the  Water 
Resources  Development  Act  (WRDA)  of  2007,  which  requires  that  studies 
for  certain  Corps  projects  undergo  independent  peer  review.5  To  conduct 
such  peer  review,  the  Corps  hires  a  contractor  to  select  a  panel  of 
independent  experts,  who  assess  the  adequacy  and  acceptability  of  the 
economic,  engineering,  and  environmental  methods,  models,  and 
analyses  used  in  a  Corps’  project  study.  Upon  completion  of  the  peer 
review,  the  Corps  is  to  consider  recommendations  from  the  review  before 
making  a  final  decision  on  the  project.6 


3National  Research  Council,  Inland  Navigation  System  Planning:  The  Upper  Mississippi 
River-lllinois  Waterway  (Washington,  D.C.:  National  Academies  Press,  2001),  and  GAO, 
Corps  of  Engineers:  Observations  on  Planning  and  Project  Management  Processes  for 
the  Civil  Works  Program,  GAO-06-529T  (Washington,  D.C.:  Mar.  15,  2006). 

4GAO-06-529T.  GAO’s  review  focused  on  four  civil  works  studies. 

5Water  Resources  Development  Act  of  2007,  Pub.  L.  No.  110-114,  §  2034  (Nov.  8,  2007), 
codified  at  33  U.S.C.  §  2343. 

6The  final  Corps  approval  of  a  project  recommendation  is  signed  by  the  Chief  of 
Engineers — the  Corps’  commanding  officer — in  what  is  known  as  the  signed  Chiefs 
report.  This  report  summarizes  the  study’s  results  and  includes  a  final  project 
recommendation;  the  report  is  generally  submitted  to  Congress  for  authorization  of 
construction. 
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You  asked  us  to  review  the  efforts  the  Corps  has  made  to  implement  and 
comply  with  the  independent  peer  review  requirements  in  section  2034  of 
WRDA  2007.  This  report  examines  (1)  the  number  of  Corps  project 
studies  that  have  undergone  independent  external  peer  review  in 
response  to  section  2034,  (2)  the  cost  of  these  peer  reviews,  (3)  the 
extent  to  which  the  Corps’  process  for  determining  if  a  project  study  is 
subject  to  peer  review  is  consistent  with  section  2034,  (4)  the  process  the 
Corps  uses  to  ensure  that  the  contractors  it  hires  and  the  experts  the 
contractors  select  to  review  project  studies  are  independent  and  free  from 
conflicts  of  interest,  and  (5)  the  extent  to  which  peer  review 
recommendations  have  been  incorporated  into  project  studies. 

To  conduct  this  work,  we  reviewed  relevant  legal  requirements,  policy 
guidance,  review  plans,  and  peer  review  reports  for  project  studies  that 
were  subject  to  a  peer  review  and  for  which  a  report  had  been  completed 
since  WRDA  2007  was  passed.  In  addition,  we  selected  a 
nongeneralizable  sample  of  six  peer  reviews  to  examine  in  greater  depth, 
to  better  understand  the  costs  associated  with  conducting  these  reviews, 
as  well  as  the  overall  impact  of  the  process  on  the  timeline  of  the  project 
study  and  the  study’s  outcome.  Because  this  sample  was  a  nonprobability 
sample,  the  information  derived  from  these  reviews  is  not  generalizable  to 
all  peer  reviews,  but  the  reviews  serve  as  illustrative  examples  that 
provide  valuable  insights  into  the  Corps’  peer  review  process;  we 
selected  a  review  from  each  planning  center  of  expertise  and  at  least  one 
for  each  of  the  three  contractors  selected  to  conduct  peer  reviews.  We 
conducted  semistructured  interviews  with  officials  from  Corps 
headquarters,  the  planning  centers  of  expertise  involved  in  managing  the 
peer  reviews,  all  of  the  Corps’  eight  divisions,  and  from  10  geographically 
dispersed  districts  that  had  conducted  project  studies  that  underwent  peer 
review.  We  also  conducted  semistructured  interviews  with  the  three 
contractors,  as  well  as  with  selected  peer  review  panel  members  and 
local  sponsors  of  Corps  civil  works  projects.7 

To  determine  the  number  of  studies  that  underwent  peer  review  and  the 
cost  of  these  reviews,  we  reviewed  all  completed  peer  review  reports  and 


7The  Corps  also  typically  receives  funds  from  each  project’s  local  sponsor,  which  may  be 
a  state,  tribal,  county,  or  local  agency  or  government.  WRDA  of  1 986  stipulated  that 
nonfederal  sponsors  share  the  cost  of  planning  and  implementing  most  Corps  civil  works 
projects.  The  division  of  federal  and  nonfederal  cost  sharing  required  varies  by  project 
purpose. 
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contract  award  documentation.  In  addition,  for  the  six  peer  reviews  we 
examined  in  depth,  we  analyzed  information  on  costs  associated  with 
managing  the  review  process,  including  those  associated  with  district  and 
other  staff  time.  To  determine  the  extent  to  which  the  Corps’  process  for 
determining  if  a  study  is  subject  to  peer  review  is  consistent  with  section 
2034,  we  analyzed  the  legal  requirements  and  relevant  Corps  policy 
guidance  for  determining  when  to  conduct  peer  reviews,  and  we  reviewed 
documentation  on  Corps  decisions.  To  determine  the  process  the  Corps 
uses  to  ensure  that  the  contractors  it  hires  and  the  experts  the  contractors 
select  are  independent  and  free  from  conflicts  of  interest,  we  analyzed 
relevant  Corps  policy  guidance  and  reviewed  documentation  provided  by 
the  contractors  to  demonstrate  that  they  and  the  selected  experts  meet 
the  requirements.  To  determine  the  extent  to  which  recommendations  are 
incorporated  into  project  studies,  we  analyzed  relevant  policy  guidance 
and  summarized  peer  review  recommendations  and  Corps  responses, 
but  we  did  not  assess  the  quality  of  the  peer  review  recommendations  or 
the  technical  sufficiency  of  the  Corps  responses  to  these 
recommendations.  In  addition,  for  the  six  reviews  examined  in  depth,  we 
analyzed  project  studies,  recommendations,  and  Corps  responses  to 
describe  the  overall  timeline  and  impact  of  the  peer  review  process. 
Appendix  I  describes  our  scope  and  methodology  in  greater  detail. 

We  conducted  this  performance  audit  from  April  201 1  to  March  2012  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


Corps  Process  for  Civil  The  purpose  of  the  Corps’  civil  works  project  study  process  is  to  inform 

Works  Project  Studies  federal  decision  makers  whether  a  water  resources  project  warrants 

further  federal  investment.  The  study  process  is  conducted  in  two  phases: 
reconnaissance  and  feasibility.  In  the  reconnaissance  phase,  the  Corps 
conducts  an  initial  evaluation  of  potential  solutions  to  a  water  resources 
problem.  If  the  Corps  determines  that  a  project  potentially  warrants 
federal  investment,  it  proceeds  to  a  more  detailed  feasibility  study.  The 
feasibility  phase  generally  begins  with  the  signing  of  a  feasibility  cost- 
share  agreement  between  the  Corps  and  the  local  project  sponsor. 
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Feasibility  studies  are  generally  prepared  by  the  Corps’  38  district  offices, 
with  review  and  oversight  provided  by  the  cognizant  Corps  division  office 
and  by  headquarters.  During  the  feasibility  phase,  the  Corps  formulates 
and  evaluates  alternative  plans  for  achieving  the  project’s  objectives  and 
reviews  the  proposed  project  to  assess  whether  the  benefits  of 
constructing  it  outweigh  its  costs.  At  the  beginning  of  this  phase,  a 
feasibility  scoping  meeting  is  held  to  bring  the  Corps,  the  local  sponsor, 
and  other  agencies  together  to  reach  agreement  on  the  problems  and 
solutions  to  be  investigated  during  the  feasibility  study  and  the  scope  of 
the  analysis  required.  The  next  step  includes  an  alternative  formulation 
briefing  to  identify  and  resolve  any  legal  or  policy  concerns  and  to  obtain 
headquarters  approval  of  the  tentatively  selected  plan  and  to  release  the 
draft  report  to  the  public.  Finally,  the  draft  feasibility  report — which 
presents  the  study  results  and  findings,  including  those  developed  in  the 
reconnaissance  phase — is  released  to  the  public.  At  the  conclusion  of  the 
feasibility  phase,  the  Corps  selects  a  recommended  plan  for  proceeding 
with  the  project. 

The  feasibility  report  also  includes  analysis  and  documentation  to  meet 
the  requirements  of  the  National  Environmental  Policy  Act  (NEPA).8 
Under  NEPA,  federal  agencies  are  to  assess  the  effects  of  major  federal 
actions,  such  as  Corps  construction  projects,  that  significantly  affect  the 
environment  and  prepare  a  detailed  statement  on  the  environmental 
impacts  of  those  actions.  NEPA  has  two  principal  purposes:  (1)  to  ensure 
that  an  agency  carefully  considers  detailed  information  concerning 
significant  environmental  impacts  and  (2)  to  ensure  that  this  information 
will  be  made  available  to  the  public.  NEPA  requires  an  agency  to  prepare 
a  detailed  statement  on  the  environmental  impacts  of  any  “major  federal 
action”  significantly  affecting  the  environment.  NEPA  implementing 
regulations  generally  require  an  agency  to  prepare  either  an 
environmental  assessment9  or  an  environmental  impact  statement 


8Pub.  L.  No.  91-190,  83  Stat.  852  (1970),  codified  as  amended  at  42  U.S.C.  §§  4321-4347 
(2011). 

9An  environmental  assessment  is  a  concise  public  document  that  provides  sufficient 
evidence  and  analysis  for  determining  whether  to  prepare  an  environmental  impact 
statement  or  a  finding  of  no  significant  impact  and  is  to  include  brief  discussions  of  the 
need  for  the  proposal,  alternatives,  the  environmental  impacts  of  the  proposed  action  and 
alternatives,  and  a  listing  of  agencies  and  persons  consulted.  40  C.F.R.  §  1508.9  (2011). 
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(EIS). 10  NEPA  implementing  regulations  also  specify  requirements  and 
procedures — such  as  providing  the  public  with  an  opportunity  to  comment 
on  the  draft  EIS  for  at  least  45  days. 

Corps  project  studies  have  historically  been  subject  to  various  levels  of 
internal  and  external  review  under  a  number  of  authorities  as  well  as  the 
Chief  of  Engineers’  responsibility  to  ensure  the  quality  of  Corps  studies. 
For  example,  in  1902,  Congress  created  the  Board  of  Engineers  for 
Rivers  and  Harbors,  which  was  the  result  of  efforts  to  address 
inconsistent  treatment  of  proposed  Corps  projects.  The  board  was  made 
up  of  Corps  staff.  Until  1992,  when  Congress  terminated  the  board,  it 
reviewed  thousands  of  Corps  studies  for  civil  works  projects  and  made 
unfavorable  recommendations  on  more  than  half.  At  the  time  the  board 
was  abolished,  there  was  concern  that  too  much  duplicative  review  was 
occurring  between  the  board  and  other  internal  Corps  review  processes. 
Subsequently,  in  the  Flood  Control  Act  of  1944,  Congress  established  a 
mechanism  for  external  review  of  Corps  projects  by  giving  the  head  of  the 
Department  of  the  Interior  and  the  governors  of  affected  states  an 
opportunity  to  comment  on  proposed  Corps  projects  before  authorization. 
Furthermore,  starting  in  1970,  under  NEPA,  environmental  impact 
statements  for  Corps  projects  were  required  to  be  sent  to  the  heads  of 
other  federal  agencies  and  governors  of  affected  states  for  comment.  As 
a  result  of  the  Flood  Control  Act  of  1970,  Congress  created  the  position  of 
Assistant  Secretary  of  the  Army  for  Civil  Works,  who  coordinates  the 
review  of  Corps  studies  with  the  Office  of  Management  and  Budget 
(OMB)  before  they  are  submitted  to  Congress. 


Increasing  Interest  in  Peer 
Review  of  Corps  Project 
Studies 


Recent  congressional  interest  in  establishing  an  independent  external 
peer  review  process  for  Corps  project  studies  began  in  the  late  1990s, 
following  a  series  of  damaging  reports  and  events,  including  allegations 
that  the  Corps  had  manipulated  information  to  justify  projects. 
Investigations  conducted  by  NAS  and  the  Army’s  Inspector  General 
identified  various  problems  with  the  Corps  internal  review  process, 


i  n 

An  EIS  is  a  more  detailed  statement  than  an  environmental  assessment.  An  EIS  must, 
among  other  things,  (1 )  describe  the  environment  that  will  be  affected,  (2)  identify 
alternatives  to  the  proposed  action  and  identify  the  agency’s  preferred  alternative,  (3) 
present  the  environmental  impacts  of  the  proposed  action  and  alternatives,  and  (4)  identify 
any  adverse  environmental  impacts  that  cannot  be  avoided  should  the  proposed  action  be 
implemented.  42  U.S.C.  §  4332(c)  (2011),  40  C.F.R.  §§  1501.4,  1508.11  (2011). 
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including  the  manipulation  of  economic  analysis  and  potential  institutional 
bias  toward  large  construction  projects.11  Around  this  time,  WRDA  2000 
required  the  Corps  to  contract  with  NAS  to  study  and  make 
recommendations  concerning  the  use  of  peer  review  for  feasibility 
reports,  including  recommending  potential  criteria  to  determine  how  to 
apply  peer  review.12  In  2002,  NAS  released  its  study  concluding  that  the 
Corps’  more  complex  water  resources  project  planning  studies  should  be 
subject  to  external,  independent  review.13  The  study  also  found  that  not 
all  Corps  project  studies  necessarily  require  such  review,  recommending 
instead  that  external  peer  review  be  reserved  for  studies  that  are 
expensive,  will  affect  a  large  area,  are  highly  controversial,  or  involve  high 
levels  of  risk.  The  study  estimated  that  about  five  Corps  projects  per  year 
would  likely  be  subject  to  this  level  of  review.  According  to  the  NAS  study, 
criteria  for  selecting  the  appropriate  level  of  review  should  balance  the 
risks  and  consequences  of  inadequate  review  against  the  resources 
required  for  more  complex  and  stringent  levels  of  review.  In  addition,  the 
study  identified  several  criteria  that  should  be  considered  in  determining 
the  appropriate  level  of  review  for  Corps  studies,  primarily  that  as  project 
magnitude  and  risks  increase,  an  increasing  degree  of  independence  and 
scope  of  review  are  warranted  (see  fig.  1). 


^National  Research  Council,  Review  of  the  U.S.  Army  Corps  of  Engineers  Restructured 
Upper  Mississippi  River-lllinois  Waterway  Feasibility  Study  (Washington,  D.C.:  National 
Academies  Press,  2004),  and  National  Research  Council,  New  Directions  in  Water 
Resources  Planning  for  the  U.S.  Army  Corps  of  Engineers  (Washington,  D.C.:  National 
Academies  Press,  1999). 

12Water  Resources  Development  Act  of  2000,  Pub.  L.  No.  106-541,  §  216(b)  (2000). 

13National  Research  Council,  Review  Procedures  for  Water  Resources  Project  Planning 
(Washington,  D.C.:  National  Academies  Press,  2002). 
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Figure  1:  National  Academy  of  Sciences  Criteria  for  Selecting  an  Appropriate  Level  of  Review 


High 

Project  magnitude 

•  Cost 

•  Benefits 

•  Complexity  Medium 

•  Scale 

•  Cumulative  effects 

•  Long-term  effects 


Low 


Review  characteristics 

•  Increasing  degree  of  independence 

•  Increasing  depth  and  complexity  of  review 

•  Increasing  scope  and  diversity  of  expertise 

•  Increasing  cost  and  duration 


Low 


Medium  High 


Project  risks 

•  Potential  for  failure  or  controversy 

•  Uncertainties  of  predictions  and  outcomes 

•  Irreversibility 


Source  Reprinted  with  permission  from  National  Academy  of  Sciences,  Review  Procedures  for  Water  Resources  Project  Planning 
(Washington,  DC.:  National  Academies  Press,  2002),  courtesy  of  the  National  Academies  Press. 


In  addition  to  recommendations  related  to  the  appropriate  level  of  review 
for  project  studies,  the  2002  NAS  study  made  several  other 
recommendations  about  the  Corps’  peer  review  process.  It  recommended 
that  peer  review  results  be  presented  to  the  Chief  of  Engineers  before  a 
final  decision  on  a  project  study  is  made,  that  the  Chief  of  Engineers 
respond  in  writing  to  each  key  point  of  the  peer  review  report,  and  that 
peer  review  be  initiated  early  enough  in  the  Corps’  study  process  so  that 
review  results  can  be  meaningfully  incorporated  into  project  design. 

After  NAS  published  its  2002  study,  OMB  in  December  2004  issued  its 
Final  Information  Quality  Bulletin  for  Peer  Review  citing  the  Information 
Quality  Act,14  as  well  as  its  general  authorities  to  oversee  the  quality  of 
agency  information,  analyses,  and  regulatory  actions.  This  OMB  bulletin 


14Pub.  L.  No.  106-554,  App.  C,  Title  V,  §  515(a)  (2000). 
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established  governmentwide  guidance  on  enhancing  peer  review 
practices  and  covers  what  information  is  subject  to  peer  review,  the 
selection  of  appropriate  peer  reviewers,  opportunities  for  public 
participation,  and  related  issues.15  The  Corps’  Engineering  Circular  1105- 
2-408  (EC  408)  was  issued  in  May  2005  and  established  procedures  for 
ensuring  the  credibility  and  quality  of  Corps  documents  by  supplementing 
its  previous  review  process,  including  to  add  external  peer  review  to  its 
review  process  in  special  cases  where  risk  and  magnitude  warrant  this 
level  of  review. 

The  Corps  faced  further  criticism  after  the  failure  of  Corps  levees  and 
floodwalls  in  New  Orleans  in  the  wake  of  Hurricane  Katrina  in  August 
2005.  In  2006,  the  Corps  announced  “Twelve  Actions  for  Change,”  which 
included  a  set  of  actions  intended  to  transform  the  Corps’  priorities, 
processes,  and  planning  and  apply  lessons  learned  from  Hurricanes 
Katrina  and  Rita.  Among  these  actions  was  to  employ  independent  review 
for  projects  with  significant  consequences,  especially  the  potential  for  loss 
of  life  if  the  project  were  to  fail. 


Section  2034  of  WRDA  of  In  November  2007,  Congress  passed  WRDA  2007  and  included  section 
2007  2034,  which  establishes  a  7-year  trial  period  for  peer  reviews  of  certain 

studies  of  civil  works  projects;  this  trial  generally  applies  to  project  studies 
initiated  by  the  Corps  from  November  2005  through  November  201 4. 16 
The  Corps  was  to  provide  an  initial  report  to  Congress  on  its 
implementation  of  the  peer  review  trial  under  section  2034  by  November 
201 0  and  is  to  provide  a  final  report  by  November  201 3.  In  February 
2011,  the  Corps  submitted  its  initial  report  to  Congress  summarizing  its 


15Office  of  Management  and  Budget,  Final  Information  Quality  Bulletin  for  Peer  Review, 
M-05-03  (Washington,  D.C.:  2004).  The  OMB  guidance  for  peer  review  applies  to 
important  scientific  assessments,  referred  to  as  “influential  scientific  information,”  which 
includes  “highly  influential  scientific  assessments,”  disseminated  by  the  federal 
government.  The  guidance  calls  for  peer  review  of  information  that  is  based  on  novel 
methods  or  presents  complex  challenges  for  interpretation,  contains  precedent-setting 
methods  or  models,  presents  conclusions  that  are  likely  to  change  prevailing  practices,  or 
is  likely  to  affect  policy  decisions  that  have  a  significant  impact. 

16Section  2034  applies  to  (1)  project  studies  initiated  from  November  2005  through 
November  2007  and  for  which  the  array  of  alternatives  to  address  the  water  resources 
problem  had  not  been  identified  and  to  (2)  project  studies  initiated  from  November  2007 
through  November  2014. 


Page  9 


GAO-12-352  Army  Corps  of  Engineers 


experiences  implementing  the  peer  review  process  in  response  to  the 
requirement  in  section  2034  of  WRDA  2007. 

Section  2034  defines  a  project  study  as  a  feasibility  or  reevaluation 
study — including  the  EIS  for  that  study — or  any  other  study  associated 
with  a  modification  of  a  water  resources  project  that  includes  an  EIS. 
Under  section  2034,  project  studies  that  meet  at  least  one  of  the  following 
criteria  are  required  to  undergo  peer  review: 

•  The  project  has  an  estimated  total  cost  of  more  than  $45  million. 

•  The  governor  of  an  affected  state  requests  an  independent  peer 
review. 

•  The  Chief  of  Engineers  determines  that  the  project  study  is 
controversial  (i.e.,  significant  public  dispute  exists  as  to  the  project’s 
size,  nature,  or  effects  or  its  economic  or  environmental  costs  or 
benefits). 

In  addition,  if  the  head  of  a  federal  or  state  agency  charged  with  reviewing 
a  project  study  determines  the  project  is  likely  to  have  a  significant 
adverse  impact  on  environmental,  cultural,  or  other  resources,  he  or  she 
may  request  that  the  Corps  consider  a  peer  review  by  an  independent 
panel  of  experts. 

WRDA  2007  also  provides  some  instances  where  exceptions  may  be 
made  to  peer  review.  For  example,  the  Corps  may  exclude  from  peer 
review  certain  projects  having  a  total  estimated  cost  of  more  than 
$45  million  but  do  not  include  an  EIS,  have  not  been  determined  by  the 
Corps  to  be  controversial,  and  come  below  specified  thresholds  of 
adverse  impacts.  The  Corps  may  also  exclude  other  project  studies 
meeting  certain  exclusion  criteria.  For  example,  the  Corps  may  exclude 
studies  that  involve 

•  only  the  rehabilitation  or  replacement  of  existing  hydropower  turbines, 
lock  structures,  or  flood  control  gates  within  the  same  footprint  and  for 
the  same  purpose  as  an  existing  water  resources  project; 

•  an  activity  for  which  the  Corps  and  industry  have  ample  experience, 
so  the  activity  may  be  considered  routine;  and 

•  minimal  life  safety  risk. 
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Section  2034  also  has  requirements  for  the  Corps  concerning  the  peer 
review  panel  and  its  independence,  as  well  as  for  timing  the  peer  review 
and  publishing  peer  review  reports,  as  described  in  more  detail  below. 


Contractor  Selection, 
Conflicts  of  Interest,  and 
Independence 


Under  section  2034,  the  Corps  is  required  to  contract  with  NAS,  a  similar 
independent  scientific  and  technical  advisory  organization,  or  an  “eligible 
organization”  to  establish  a  panel  of  experts  that  will  review  a  project 
study.  Section  2034  defines  an  eligible  organization  as  having  the 
following  five  characteristics: 


•  is  a  501(c)(3)  tax-exempt  organization, 

•  is  independent, 

•  is  free  from  conflicts  of  interest, 

•  does  not  carry  out  or  advocate  for  or  against  federal  water  resources 
projects,  and 

•  has  experience  in  establishing  and  administering  panels. 


Section  2034  states  that  when  establishing  peer  review  panels, 
contractors  must  apply  NAS’s  policy  for  selecting  committee  members  to 
ensure  that  they  also  have  no  conflicts  of  interest.  The  NAS  Policy  on 
Committee  Composition  and  Balance  and  Conflicts  of  Interest  outlines 
several  criteria  for  selecting  peer  review  panel  members,  including  the 
following: 


•  All  panel  members  must  be  highly  qualified  in  terms  of  knowledge, 
training,  and  experience. 

•  The  knowledge,  experience,  and  perspectives  of  the  panel  members 
must  be  thoughtfully  and  carefully  assessed  and  balanced  in  terms  of 
the  subtleties  and  complexities  of  the  particular  scientific,  technical, 
and  other  issues  to  be  addressed. 

•  Potential  sources  of  bias  must  be  assessed  to  determine  that  the 
panel’s  report  will  not  be  compromised  by  issues  of  bias  or  lack  of 
objectivity. 
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•  Panel  members  must  not  have  financial  interests  that  could 
significantly  impair  their  objectivity  or  create  an  unfair  competitive 
advantage  for  any  person  or  organization. 

•  Panel  members  must  not  obtain  and  use,  or  intend  to  use,  confidential 
information  not  reasonably  available  to  the  public  for  their  own  or 
direct  and  substantial  economic  benefit. 

•  Panel  members  must  not  serve  as  a  member  on  a  peer  review  panel 
that  is  to  review  the  panel  member’s  own  work. 

•  Panel  members  must  not  have  become  committed  to  a  fixed  position 
related  to  the  review  for  which  they  have  a  significant  directly  related 
interest  or  duty. 

•  Persons  currently  employed  by  the  agency  sponsoring  the  study 
cannot  be  panel  members,  except  in  extremely  limited  special 
circumstances.17 

Additionally,  section  2034  requires  that  both  the  experts  selected  for  the 
peer  review  panels  and  the  organizations  managing  the  peer  review 
selections  be  independent.  Section  2034  does  not  define  the  term 
independent,  but  both  the  2002  NAS  peer  review  study  and  OMB’s  Final 
Information  Quality  Bulletin  for  Peer  Review  regard  independent  to  mean 
external  to  the  Corps.18  Specifically,  the  NAS  study  states  that  a  fully 
independent  review  can  be  accomplished  only  by  reviewers  who  are  free 
of  conflicts  of  interest  and  are  appointed  by  a  group  external  to  the  Corps. 
Similarly,  the  OMB  bulletin  states  that  independent  reviewers  are 
generally  not  employed  by  the  agency  or  office  producing  the  document. 


17ln  special  circumstances  and  to  the  extent  not  prohibited  by  federal  or  state  laws  or 
regulations,  such  an  individual  may  serve  as  a  member  of  such  a  committee  where  the 
following  requirements  are  met:  (1)  the  service  of  the  individual  on  the  committee  must  be 
based  upon  the  unique  scientific  or  technical  expertise  the  individual  brings  to  the 
committee;  (2)  the  individual  must  not  be  involved  in  any  way  within  the  agency  in  any 
deliberative  or  decision-making  process  or  any  policymaking  or  similar  process  relating  to 
the  study  or  other  activity  or  the  expected  or  intended  results  of  the  study  or  other  activity; 
and  (3)  it  must  be  specifically  determined  during  the  committee  appointment  process  that 
service  by  the  individual  will  not  compromise,  or  appear  to  compromise,  the  independence 
or  objectivity  of  the  particular  study  or  other  activity  in  which  the  committee  is  engaged. 

18National  Research  Council,  Review  Procedures  (2002),  and  Office  of  Management  and 
Budget,  Final  Information  Quality  Bulletin  for  Peer  Review  (2004). 
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Timing  of  Peer  Review  and 
Publishing  of  Peer  Review 
Reports 


Section  2034  requires  that  the  peer  review  be  conducted  during  the 
period  from  the  signing  of  the  feasibility  cost-share  agreement  between 
the  Corps  and  the  local  sponsor  and  60  days  after  the  last  day  of  the 
public  comment  period  for  the  draft  project  study.  Additionally,  section 
2034  lists  three  points  during  the  feasibility  study  process  at  each  of 
which  the  Chief  of  Engineers  must  consider  whether  to  initiate  peer 
review: 

•  when  the  without-project  conditions — current  and  forecasted 
conditions  if  the  project  were  not  constructed — are  identified, 

•  when  the  array  of  alternatives  to  be  considered  are  identified,  and 

•  when  the  preferred  alternative  is  identified. 

Figure  2  shows  the  key  steps  in  the  feasibility  study  process,  including 
those  specified  in  section  2034.  The  Corps  can  conduct  peer  review  at 
any  time  during  the  steps  shown  highlighted  in  gray  in  the  figure,  but 
according  to  Corps  officials  it  generally  conducts  peer  review  after  the 
draft  feasibility  report  has  been  completed.  The  Washington-level  review 
shown  as  the  final  step  in  the  figure  concludes  with  a  signed  Chiefs 
report  for  project  studies  that  will  be  submitted  to  Congress  for 
authorization. 
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Figure  2:  Key  Steps  in  the  Feasibility  Study  Process 


Source:  GAO  analysis  of  Corps  documents. 

Note:  The  Corps’  independent  peer  review  process  can  take  place  at  any  time  during  the  steps 
enclosed  by  the  shaded  gray  area. 

aThe  feasibility  scoping  meeting  brings  the  Corps,  local  sponsor  and  relevant  government  agencies 
together  to  reach  agreement  on  the  problems  and  solutions  to  be  investigated  during  the  feasibility 
study.  The  without-project  conditions  are  identified  during  the  feasibility  scoping  meeting. 

bThe  alternative  formulation  briefing  confirms  that  the  plan  formulated  for  the  project  study  and  the 
definition  of  federal  and  nonfederal  responsibilities  are  consistent  with  applicable  laws,  statutes, 
executive  orders,  regulations,  and  current  policy  guidance.  The  array  of  alternatives  to  be  considered 
is  identified  at  the  alternative  formulation  briefing. 

cThe  preferred  alternative  is  identified  when  the  draft  feasibility  report  is  released. 
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Section  2034  also  requires  the  Corps  to  prepare  and  make  publicly 
available  a  written  response  to  all  completed  peer  review  reports  before  it 
finalizes  project  studies.  The  Corps  must  provide  Congress  with  a  copy  of 
both  the  completed  peer  review  report  and  the  Corps’  written  response 
when  the  signed  Chief’s  report  or  other  final  decision  document  for  the 
project  study  is  transmitted  to  Congress. 


Corps  Guidance  for  The  Corps’  Engineering  Circular  1165-2-209  (EC  209)  was  issued  in 

Implementing  Peer  Review  January  2010  and  establishes  its  civil  works  review  policy,  which  outlines 

the  processes  for  implementing  product  review  requirements  for  Corps 
civil  works  projects.  EC  209  was  developed  to  include  the  specific 
requirements  for  independent  peer  review  contained  in  section  2034, 
OMB’s  2004  peer  review  guidance,  as  well  as  other  Corps  policy 
considerations.  EC  209  requires  that  districts,  in  coordination  with  the 
relevant  Corps  planning  center  of  expertise,  prepare  review  plans  for 
project  studies.  These  review  plans  are  to  describe  the  appropriate  levels 
of  potential  review  that  the  specific  project  study  will  be  subject  to,  such 
as  the  district’s  quality  control  procedures,  agency  technical  review,  peer 
review,  and  policy  and  legal  review.  If  a  project  study  review  plan 
indicates  that  a  peer  review  will  not  be  conducted,  then  the  district  is 
required  to  develop  a  risk-based  recommendation  for  why  the  peer  review 
is  not  required.  This  recommendation  should  document,  among  other 
things,  that  the  project  study  is  of  such  limited  scope  or  impact  that  it 
would  not  benefit  from  a  peer  review.19 


It  Is  Unclear  How 
Many  Peer  Reviews 
Have  Been  Completed 
in  Response  to 
Section  2034 
Requirements 


Since  enactment  of  WRDA  2007,  49  Corps  civil  works  project  studies 
have  undergone  peer  review  as  of  January  2012,  but  it  is  unclear  how 
many  of  these  reviews  were  performed  in  response  to  the  requirements  in 
section  2034.  This  is  because  the  Corps  does  not  make  specific 
determinations  or  track  whether  a  peer  review  is  being  conducted  in 
response  to  the  requirements  of  section  2034.  Of  the  49  project  studies 
that  underwent  peer  review,  the  majority  were  for  ecosystem  restoration 
projects,  flood  risk  management  projects,  or  deep  draft  navigation 
projects.  (App.  II  lists  the  49  project  studies  that  underwent  peer  review 
since  WRDA  2007  was  passed,  including  information  on  project  and 


1 Q 

Earlier  Corps  guidance  related  to  peer  review  was  issued  in  August  2008  in  Engineering 
Circular  1105-2-410  (EC  410),  Review  of  Decision  Documents,  and  in  May  2005  in 
Engineering  Circular  1105-2-408  (EC  408),  Peer  Review  of  Decision  Documents. 


Page  15 


GAO-12-352  Army  Corps  of  Engineers 


study  type,  as  well  as  the  district,  division,  and  planning  center  of 
expertise  associated  with  each  study.) 

Moreover,  it  is  not  possible  to  determine  how  many  project  studies  were 
required  to  undergo  peer  review  in  response  to  WRDA’s  section  2034 
requirements  because  the  Corps  does  not  centrally  track  project  studies 
and  could  not  provide  us  a  list  of  all  project  studies  that  fell  within  the 
scope  of  section  2034.  Our  review  of  relevant  Corps  documents  for  the  49 
project  studies  that  underwent  peer  review,  such  as  review  plans  and 
completed  review  reports,  found  that  none  of  these  documents  specifies 
the  authority  under  which  peer  reviews  were  conducted.  Corps 
headquarters  officials  told  us  that  the  Corps  does  not  make  specific 
determinations  as  to  whether  a  peer  review  is  being  conducted  under 
section  2034  but  instead  focuses  on  ensuring  that  the  peer  review  is 
being  carried  out  in  compliance  with  EC  209,  which,  in  their  view, 
complies  with  section  2034.  These  officials  also  told  us  that,  to  ensure  the 
quality  of  Corps  project  studies,  the  agency  may  choose  to  conduct  peer 
reviews  under  its  other  authorities,  even  if  those  peer  reviews  are  not 
required  by  section  2034. 

In  February  201 1 ,  the  Corps  submitted  its  initial  report  to  Congress  in 
response  to  the  requirement  in  section  2034  of  WRDA  2007  summarizing 
its  experiences  implementing  the  peer  review  process.  In  the  report,  the 
Corps  noted  that  the  29  peer  reviews  that  had  been  completed  as  of 
February  2011  followed  the  procedures  described  in  agency  and  OMB 
guidance.  The  Corps  report  stated  that,  in  its  view,  section  2034 
provisions  reinforce  and  add  further  definition  to  the  Corps’  process. 
Nevertheless,  because  the  Corps  did  not  distinguish  which  studies  had 
been  selected  for  peer  review  in  accordance  with  section  2034,  we 
believe  that  it  did  not  provide  Congress  with  the  type  of  information 
required  by  section  2034  that  would  help  congressional  decision  makers 
evaluate  the  trial  program. 


Completed  Peer 
Reviews  Have  Cost 
Millions  of  Dollars  in 
Direct  and  Indirect 
Costs 


The  49  peer  reviews  conducted  by  the  Corps  since  November  2007 
resulted  in  direct  costs  of  about  $9  million  in  contract  costs  and  contract 
administration  fees.  In  addition,  Corps  staff  resources  were  also  used  to 
manage  peer  reviews,  but  these  costs  are  not  fully  quantifiable. 
Furthermore,  the  addition  of  peer  review  to  the  Corps  study  process  has 
resulted  in  indirect  costs  by  altering  project  study  schedules  because  of 
the  additional  time  required  to  complete  the  peer  review.  In  some  cases 
where  a  peer  review  was  not  planned  for  during  the  early  stages  of  the 
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study  process,  significant  delays  in  the  project  studies  have  resulted  from 
the  addition  of  the  peer  review. 


Direct  Costs  Related  to  The  49  project  studies  for  which  the  Corps  completed  peer  reviews  since 

Peer  Reviews  Have  Totaled  November  2007  cost  about  $9  million  in  contract  costs  and  contract 
Over  $9  Million  administration  fees  to  establish  and  manage  the  expert  panels  for  these 

reviews.  In  addition,  Corps  staff  resources  were  also  used  to  manage  the 
peer  reviews,  but  these  costs  are  not  fully  quantifiable.  The  Corps  used 
the  services  of  three  contractors  to  manage  the  peer  review  process:  the 
nonprofit  Battelle  Memorial  Institute,  which  managed  46  of  the  reviews; 
the  nonprofit  Noblis,  which  managed  two;  and  NAS,  which  managed  one. 
The  cost  per  panel  varied  considerably.  For  example,  the  contracts 
managed  by  Battelle  cost  from  about  $76,000  to  $484,000  for  studies  that 
underwent  peer  review,20  but  the  panel  managed  by  NAS  cost  over 
$500,000.  (See  app.  II  for  information  on  the  contract  costs  for  each  of 
the  49  peer  reviews.)  In  addition  to  the  $9  million  in  contractor  costs,  the 
Corps  incurred  about  $109,000  for  the  administration  of  the  contracts  for 
the  49  peer  reviews.  Specifically,  the  Corps  used  two  different  entities — 
the  Institute  for  Water  Resources  and  the  Army  Research  Office — to 
administer  these  contracts,  and  both  of  these  entities  charged 
administration  fees.21  These  fees  ranged  from  no  fee  to  3  percent  of  the 
contract  cost. 

Corps  staff  resources  were  also  used  to  manage  the  peer  review 
process — including  developing  the  scope  of  work  for  reviews, 
coordinating  establishment  of  contracts,  reviewing  contract  proposals, 
and  responding  to  panel  comments  that  the  Corps  received  during  a  peer 
review  process.  Corps  district,  division,  headquarters,  and  planning- 
center-of-expertise  staff  also  spent  time  managing  the  peer  review 
process.  The  total  of  these  costs,  however,  is  not  fully  quantifiable  across 


on 

In  some  cases,  individual  project  studies  underwent  multiple  phases  of  peer  review 
because  of  revisions  made  to  the  study  after  the  initial  peer  review  report,  and  peer  review 
was  completed  under  multiple  reports  and  contracts.  This  dollar  range  does  not  include 
one  contract  that  cost  about  $677,000  because  the  cost  covered  peer  review  for  five 
studies. 

21The  Institute  for  Water  Resources  is  the  Corps  center  of  expertise  for  integrated  water 
resources  management.  The  U.S.  Army  Research  Laboratory's  Army  Research  Office  is 
the  Army's  research  agency  in  the  engineering,  physical,  information,  and  life  sciences. 
These  entities’  contracting  offices  administer  the  Corps’  contracts  for  establishing  and 
managing  peer  review  panels. 


Page  17 


GAO-12-352  Army  Corps  of  Engineers 


all  Corps  districts  because  not  all  districts  track  district  staff  time  spent  on 
peer  reviews.  For  those  districts  that  did  track  or  could  estimate  district 
staff  time  spent  on  peer  review-related  activities,  we  found  the  following 
examples  of  the  staff  resources  that  may  have  been  dedicated  to 
managing  and  responding  to  peer  review  activities: 

•  The  Green  Bay  Dredged  Material  Management  Plan  peer  review  cost 
about  $101,000  in  staff  time,  according  to  data  provided  by  district 
officials.  But  district  officials  involved  in  this  peer  review  said  that  the 
cost  in  terms  of  staff  time  may  have  been  higher  than  typical  because 
this  peer  review  was  the  first  conducted  for  a  study  in  that  district. 

•  The  Chatfield  Water  Reallocation  Study  peer  review  cost  about 
$20,000  in  staff  time  as  of  December  201 1 ,  but  the  Corps  response 
has  not  been  completed  for  this  peer  review,  and  additional  staff  time 
could  be  involved. 

•  For  the  Boston  Harbor  study,  district  officials  estimated  that  costs 
totaled  about  $77,000  for  district  staff  time,  agency  technical  review 
team  labor,  and  contractor  fees  for  assisting  the  district  with 
responding  to  peer  review  panel  comments. 

•  For  the  American  River  study,  district  officials  estimated  that  costs 
came  to  about  $40,000  for  district  staff  time. 

Similar  to  district  staff  time,  other  staff  time  involved  in  managing  the  peer 
review  process,  including  headquarters,  division,  and  some  planning 
centers  of  expertise  time,  is  also  not  always  tracked  and  therefore  not 
fully  quantifiable:  some  of  these  positions  are  funded  with  general  funds 
and  not  project-specific  funds,  according  to  Corps  officials  we  spoke  with. 
We  did  find  two  examples,  however,  where  planning  centers  of  expertise 
staff  time  devoted  to  peer  review  related  activities  was  tracked.  In  these 
two  instances,  the  cost  of  planning  centers  of  expertise  staff  time  devoted 
to  peer  review  activities  amounted  to  about  $12,000  for  the  peer  review  of 
the  Boston  Harbor  study  and  about  $32,000  for  the  American  River  study. 


Addition  of  Peer  Review  The  addition  of  peer  review  to  the  Corps  study  process  has  also  had  an 
Has  Affected  Project  Study  indirect  cost  because  it  has  affected  project  study  schedules.  Planning 
Schedules  centers  of  expertise  and  district  officials  estimate  that  obtaining  the 

contract  and  executing  the  peer  review  generally  take  about  a  year.  The 
breakdown  for  the  peer  review  process,  according  to  some  of  these 
officials,  is  about  3  months  to  initiate  the  contract;  3  to  6  months  for  the 
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review  to  be  completed;  and  an  additional  3  months  to  close  out  the 
review,  which  involves  responding  to  and  receiving  clarification  on  panel 
comments.  Some  of  these  processes  occur  concurrently  with  other 
aspects  of  the  project  study,  but  some  parts  of  the  peer  review  process, 
such  as  responding  to  panel  comments,  may  add  time  to  the  study 
schedule.  According  to  Corps  officials,  the  addition  of  peer  review  to  the 
project  schedule  adds  steps  to  the  review  process  and  takes  time  away 
from  other  projects.  In  addition,  according  to  several  Corps  officials,  some 
project  studies  have  been  delayed  because  the  district  did  not  allocate 
funding  for  the  peer  review  and  therefore  had  to  wait  until  additional 
funding  was  available.  In  some  cases,  this  delay  added  significant  time  to 
the  schedule.  In  contrast,  according  to  some  Corps  division  and  planning- 
center-of-expertise  officials,  when  the  project  manager  had  built  in  time 
for  the  peer  review  and  had  identified  funding  for  it  early,  the  peer  review 
process  had  much  less  of  an  impact  on  the  overall  project  study 
schedule. 

Local  sponsors  are  also  concerned  about  the  impact  that  this  additional 
time  is  adding  to  project  studies,  according  to  Corps  officials  and  local 
sponsors  we  spoke  with.  Their  concern  arises  largely  because  local 
sponsors  share  the  cost  of  the  Corps  study  and  depend  on  its  timely 
completion.  District  officials  told  us  that  because  of  the  cost-sharing 
requirements  and  the  current  economic  environment  demand  is  greater 
from  local  sponsors  for  the  Corps  to  finish  studies  quickly  and  keep  costs 
down.  Two  local  sponsors  told  us  that  delays  negatively  affect  local 
sponsors  because  they  can  lose  business  if  a  project  is  not  completed  in 
a  timely  manner.  Similarly,  sponsors  are  accountable  to  their  own  local 
governments  or  state  legislatures,  and  additional  delays  or  time  required 
for  peer  review  can  create  challenges  in  getting  continued  support  for  a 
project.  For  example,  in  the  case  of  the  Green  Bay  Dredged  Material 
Management  Plan,  Corps  officials  told  us  that  peer  review  increased  the 
cost  of  the  project  and  caused  a  5-  to  6-month  delay  at  a  time  when  the 
local  sponsor  was  attempting  to  acquire  grant  money  contingent  on 
completion  of  the  dredged  material  management  plan. 
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The  Corps’  Process 
for  Initiating  Peer 
Review  Is  More 
Expansive  and  Less 
Flexible  Than  Section 
2034  Requirements 


The  Corps’  process  for  determining  whether  a  project  study  is  subject  to 
peer  review  is  more  expansive  than  section  2034  requirements  because  it 
uses  broader  criteria;  this  has  resulted  in  peer  reviews  of  studies  that  are 
outside  the  scope  of  section  2034.  In  addition,  the  process  the  Corps 
uses  does  not  include  the  flexibility  provided  in  section  2034  to  exclude 
certain  project  studies  from  peer  review.  Moreover,  some  studies  are 
undergoing  peer  review  that  do  not  warrant  it,  according  to  some  Corps 
officials  we  spoke  with. 


The  Corps’  Process  for 
Determining  the  Need  for 
Peer  Review  Is  More 
Expansive  Than  Section 
2034  Requirements 


The  Corps’  process  for  determining  whether  a  project  study  is  subject  to 
peer  review  uses  criteria  that  are  broader  than  the  requirements  of 
section  2034.  As  table  1  shows,  the  Corps  relies  on  its  guidance  outlined 
in  EC  209  when  selecting  project  studies  for  peer  review,  and  this 
guidance  extends  beyond  section  2034  requirements. 


Table  1:  Corps  EC  209  Guidance  on  Conducting  Peer  Review 

Basis 

Criteria  for  conducting  peer  review 

Section 

2034 

OMB 

bulletin  Other3 

Significant  threat  to  human  lifeb 

X 

The  total  project  cost  is  greater  than  $45  million 

X 

Request  by  the  governor  of  an  affected  state 

X 

Request  by  the  head  of  a  federal  or  state  agency 

x 

Significant  public  dispute  as  to  size,  nature,  or 
effects  of  the  project 

xc 

Significant  public  dispute  as  to  the  economic  or 
environmental  cost  or  benefit  of  the  project 

xc 

Information  is  based  on  novel  methods 

X 

Presents  complex  challenges  for  interpretation 

x 

Contains  precedent-setting  methods  or  models 

x 

Presents  conclusions  that  are  likely  to  change 
prevailing  practices 

x 

Any  other  circumstance  where  the  Chief  of 
Engineers  determines  peer  review  is  warranted 

x 

Source:  GAO  analysis  of  Corps  guidance. 


aFor  example,  Corps  officials  noted  that  the  criteria  for  peer  review  reflect  the  agency’s  general 
authority  to  conduct  peer  review  in  association  with  its  responsibility  to  produce  quality  projects. 

bSection  2035  of  WRDA  2007  requires  that  certain  projects  in  design  or  under  construction  undergo 
independent  peer  review  in  cases  where  the  Chief  of  Engineers  determines  a  safety  assurance 
review  is  necessary,  such  as  where  failure  of  a  project  would  pose  a  significant  threat  to  human  life. 
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cSection  2034  criteria  include  that  the  Chief  of  Engineers  determines  that  the  study  is  controversial 
considering  whether  (a)  significant  public  dispute  exists  as  to  the  size,  nature,  or  effects  of  the  project; 
or  (b)  significant  public  dispute  exists  as  to  the  economic  or  environmental  costs  or  benefits  of  the 
project. 

Consequently,  the  Corps  has  selected  some  studies  for  review  based  in 
part  on  criteria  included  in  EC  209  that  are  not  required  by  section  2034, 
and  others  that  are  outside  the  scope  of  section  2034.  For  example, 
according  to  our  analysis  of  review  plans  for  44  peer  reviewed  project 
studies,22  over  one-third  identified  criteria  that  related  to  both  section 
2034  and  other  authorities.  In  addition,  the  Corps  process  for  determining 
whether  a  peer  review  is  required  has  resulted  in  30  project  studies 
undergoing  peer  review  that  were  outside  the  time  parameters  identified 
in  section  2034.  Based  on  our  analysis  of  the  characteristics  of  these 
studies,  the  Corps’  process  was  applied  to  all  studies  and  reports 
regardless  of  when  they  were  initiated,23  whereas  section  2034  applies  to 
project  studies  initiated  from  November  2005  through  November  2014. 
Specifically,  section  2034  applies  to  (1)  project  studies  initiated  from 
November  2005  through  November  2007  and  for  which  the  array  of 
alternatives  had  not  been  identified,  and  (2)  project  studies  initiated  from 
November  2007  through  November  2014.  As  a  result,  over  half  (30  of  49) 
of  the  peer  reviews  conducted  since  the  enactment  of  WRDA  2007  were 
for  project  studies  that  did  not  fall  under  the  scope  of  section  2034 
because  the  studies  were  initiated  before  November  2005. 24 

Another  reason  the  Corps’  process  for  selecting  studies  for  peer  review  is 
more  expansive  than  the  scope  of  section  2034  is  that  Corps’  guidance 


220f  the  49  peer  reviews  that  the  Corps  conducted,  4  review  plans  for  studies  that 
underwent  peer  review  were  not  updated  to  reflect  the  decision  to  conduct  peer  review 
and  indicate  that  peer  review  was  not  required,  and  1  project  study  that  underwent  peer 
review  did  not  have  a  review  plan  completed,  so  our  analysis  was  based  on  44  review 
plans. 

no 

EC  209  includes  an  exception  for  cases  where  the  final  decision  document  package  had 
been  forwarded  to  headquarters  before  August  22,  2008. 

24We  requested  a  list  of  studies  that  came  within  the  scope  of  section  2034  but  the  Corps 
does  not  track  this  information,  however,  and  it  requested  this  information  from  each  of  the 
districts.  Through  this  effort,  the  Corps  identified  134  studies  with  the  caveat  that  the  list 
was  not  complete  and  1 8  of  the  49  studies  that  underwent  peer  review  were  included  on 
this  list.  In  addition,  we  made  our  initial  request  for  this  list  of  studies  in  June  and  did  not 
receive  it  until  mid-December.  Given  the  delay  and  the  list’s  incompleteness,  we  did  not 
evaluate  the  other  1 16  studies  on  the  list  for  whether  peer  review  is  required  or  planned. 
We  note  it  is  likely  that  some  of  the  116  studies  may  not  yet  have  review  plans  nor 
decisions  on  whether  peer  review  is  to  be  conducted. 
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does  not  clearly  define  “project  study.”  The  guidance  refers  to  a  wide 
range  of  project  studies,  decision  documents,  and  work  products  that  may 
be  subject  to  peer  review,  whereas  section  2034  defines  a  project  study 
subject  to  peer  review  as  a  feasibility  or  reevaluation  study,  including  the 
EIS,  or  any  other  study  associated  with  the  modification  of  a  water 
resources  project  that  includes  an  EIS.  According  to  our  analysis  of  the 
49  studies  that  underwent  peer  review,  some  of  these  studies  did  not  fit 
this  definition.  Specifically,  34  of  the  49  studies  that  underwent  peer 
review  were  feasibility  or  reevaluation  studies  which  are  project  studies  as 
defined  by  section  2034  requirements,  8  were  other  kinds  of  reports  that 
included  an  EIS  and  therefore  may  have  been  subject  to  section  2034 
requirements,  and  7  were  neither  feasibility  nor  reevaluation  studies  and 
did  not  include  an  EIS  and  therefore  did  not  fit  the  definition  of  a  project 
study  subject  to  peer  review  under  section  2034.  For  more  details  on 
each  of  the  studies  that  underwent  peer  review,  see  appendix  II. 


The  Corps  Does  Not  Use 
the  Flexibility  in  Section 
2034  to  Exclude  Studies 
from  Review 


The  Corps’  process  for  determining  whether  peer  review  is  required  for 
project  studies  does  not  include  the  flexibility  provided  in  section  2034  to 
exclude  certain  project  studies  from  otherwise  mandatory  peer  review.  EC 
209  states  that  most  studies  should  undergo  peer  review,  and  the  Corps’ 
process  requires  that  for  any  decision  document  to  forgo  a  peer  review, 
an  exclusion  must  be  requested  and  approved  by  headquarters.  In 
addition,  guidance  provided  to  Corps  staff  on  how  to  implement  EC  209 
discourages  requests  for  exclusions,  noting  that  time  should  not  be 
wasted  shopping  around  for  exclusion  requests.  Furthermore,  agency 
guidance  and  Corps  headquarters  officials,  including  the  Director  of  Civil 
Works,  highlight  the  value  and  importance  of  peer  review  in  achieving  the 
agency’s  mission,  noting  that  an  extra  set  of  eyes  is  beneficial.  In 
addition,  Corps  headquarters  officials  told  us,  and  agency  guidance 
highlights  language  from  the  WRDA  2007  conference  report,  that 
“[sjection  2034  permits  the  Chief  of  Engineers  to  exclude  a  very  limited 
number  of  project  studies  from  independent  peer  review.”25  We  believe, 
however,  that  the  Corps  has  misconstrued  this  statement  and  overstated 
its  significance.  This  statement  is  part  of  the  explanation  of  the  exclusion 
paragraph  (a)(5),  and  does  not  apply  to  the  provision  as  a  whole; 
therefore,  this  statement  pertains  to  how  many  studies  for  which  peer 


25H.R.  Conf.  Rep.  110-280,  at  267. 
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review  is  mandatory  would  be  eligible  for  exclusion.26  Furthermore, 
another  relevant  statement  in  a  House  committee  report  on  WRDA  2007 
suggests  that  26  studies  over  7  years,  or  about  4  studies  per  year,  would 
be  expected  to  be  subject  to  peer  review.  Additionally,  the  2002  NAS 
study — which  is  prominently  mentioned  throughout  the  subsequent 
legislative  history  of  WRDA  2007 — states  that  not  all  Corps  water 
resources  project  planning  studies  will  require  external,  independent 
review,  but  the  Corps  should  institute  external  review  for  studies  that  are 
expensive,  that  will  affect  a  large  area,  that  are  highly  controversial,  or 
that  involve  high  levels  of  risk. 

In  reviewing  the  exclusion  requests  that  it  receives,  Corps  headquarters 
determines  whether  the  studies  meet  any  of  the  mandatory  requirements 
in  EC  209  for  undergoing  peer  review.  Specifically,  the  Corps  reviews 
whether  the  project  has  a  cost  estimate  of  greater  than  $45  million, 
represents  a  threat  to  health  and  safety,  is  controversial,  and  has  had  a 
request  for  peer  review  from  a  governor  or  the  head  of  a  federal  or  state 
agency.  If  studies  do  not  meet  any  of  these  criteria,  the  Corps  generally 
approves  the  study  for  exclusion  from  peer  review.  From  our  review  of  50 
studies  that  had  requested  exclusion  from  peer  review  between  2009  and 
201 1 ,  we  found  that  the  Corps  had  granted  an  exclusion  for  37  studies 
because  they  did  not  meet  any  of  the  criteria  in  EC  209  for  studies  that 
must  undergo  peer  review  except  for  one  study,  which  did  not  fit  the 
definition  of  a  project  study  in  section  2034. 27 

Under  section  2034,  however,  the  Corps  also  has  the  flexibility  to  exclude 
studies  from  peer  review  that  exceed  the  $45  million  threshold  if  they:  do 
not  have  an  EIS;  are  not  controversial;  are  expected  to  have  negligible 
adverse  impacts  on  scarce  or  unique  cultural,  historic,  or  tribal  resources; 
have  no  substantial  impacts  on  fish  and  wildlife  species  or  their  habitats; 
and  have  no  more  than  negligible  impacts  on  threatened  or  endangered 
species  or  their  critical  habitat.  Similarly,  under  section  2034,  the  Corps 


26H.  Rep.  110-80,  at  132  (discussing  H.R.  1495  §2037)  (Mar.  29,  2007). 

27This  study  exceeded  the  threshold  of  $45  million,  but  Corps  headquarters’  approval  of 
the  request  for  exclusion  said  it  was  excluded  because  it  did  not  meet  any  of  the  other 
mandatory  criteria  for  peer  review.  It  was  not  a  feasibility  study  or  a  reevaluation  study  and 
did  not  include  an  EIS.  Of  the  50  studies  that  went  through  the  exclusion  process  as  of 
November  201 1 ,  the  Corps  excluded  37.  The  Corps  decided  not  to  exclude  7  of  these 
studies  and  determined  that  4  did  not  need  to  request  an  exclusion.  A  decision  is  pending 
for  1  study  and  has  been  deferred  for  another  study. 
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may  exclude  studies  from  peer  review  that  involve  (1)  only  the 
rehabilitation  or  replacement  of  existing  hydropower  turbines,  lock 
structures,  or  flood  control  gates  within  the  same  footprint  and  for  the 
same  purpose  as  an  existing  water  resources  project;  (2)  an  activity  for 
which  there  is  ample  experience  within  the  Corps  and  industry  to  treat  the 
activity  as  being  routine;  and  (3)  minimal  risk  to  human  life  and  safety. 
Nevertheless,  according  to  our  analysis  of  exclusion  request  documents 
and  headquarters’  responses  to  these  requests,  as  of  November  201 1 , 
the  Corps  had  not  granted  an  exclusion  based  on  any  of  the  flexibilities 
included  in  section  2034. 

Several  Corps  officials  expressed  concerns  about  the  Corps  exclusion 
process.  Specifically,  some  officials  told  us  that  they  were  concerned 
about  the  cost  and  time  involved  and  said  that  the  exclusion  of  projects 
that  do  not  meet  any  of  the  mandatory  criteria  should  be  delegated  to  the 
division  offices.  In  their  opinion  such  delegation  would  help  streamline  the 
process.  Moreover,  some  of  the  studies  that  underwent  or  are  currently 
undergoing  peer  review  did  not  warrant  it,  according  to  some  Corps 
officials  we  spoke  with.  Specifically,  we  found  the  following  examples  of 
studies  that  may  not  have  warranted  a  peer  review: 

•  Two  dredged  material  management  plans  underwent  peer  review.  For 
example,  the  Green  Bay  Dredged  Material  Management  Plan 
underwent  peer  review  but  is  not  a  project  study  as  defined  by  section 
2034.  Officials  we  spoke  with  said  that  such  plans  should  not 
generally  require  peer  review  because  any  significant  impacts  would 
be  addressed  under  NEPA  and  because  the  Corps  has  sufficient 
expertise  in  the  area  of  dredging. 

•  The  Chacon  Creek  study  in  southern  Texas  underwent  peer  review 
but  should  not  have,  according  to  some  Corps  officials  we  spoke  with. 
This  study  was  for  a  project  that  would  remove  houses  from  a 
floodplain,  but  officials  said  it  should  not  require  peer  review  because 
there  are  no  structural  components  and  it  did  not  exceed  the  $45 
million  threshold.  Corps  headquarters  denied  the  request  for  exclusion 
and  stated  that  flood  studies  warrant  peer  review  because  of  the 
nature  of  the  hazard  and  the  need  to  assess  the  extent  and  treatment 
of  risk.  Headquarters  officials  highlighted  the  importance  of  assessing 
and  addressing  such  risks  in  light  of  Hurricane  Katrina  and  said  that 
flood  studies  such  as  Chacon  Creek  require  peer  review  because  of 
the  importance  of  assessing  and  decreasing  risks  associated  with 
flooding. 
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•  The  Yuba  River  General  Reevaluation  study  is  undergoing  peer 
review  but  does  not  warrant  it  according  to  some  Corps  officials. 
District  officials  told  us  that  the  study  does  not  warrant  peer  review 
because  the  construction  work  involved  has  already  been  completed, 
and  the  purpose  of  the  study  is  to  determine  the  amount  the  local 
sponsor  should  be  reimbursed  by  the  Corps. 

Officials  from  several  districts,  divisions,  and  planning  centers  of  expertise 
we  spoke  with  told  us  that  peer  review  should  be  focused  on  larger  and 
more  complex  or  controversial  projects  and  should  not  be  the  default 
approach.  Two  Corps  officials  described  the  Corps’  peer  review  policy  as 
a  one-size-fits-all  approach,  and  one  of  these  officials  stated  that  it  is 
inflexible  and  risk  averse. 


Gaps  Exist  in  the 
Corps’  Process  for 
Screening  Its 
Contractors  Who  Are 
Responsible  for 
Selecting  Experts  for 
Peer  Review  Panels 


The  Corps  has  a  process  to  review  general  information  on  contractors’ 
conflicts  of  interest  and  independence  during  its  contractor  selection 
process,  but  it  does  not  have  a  process  for  reviewing  project-specific 
information  provided  by  contractors  to  determine  if  conflicts  of  interest 
and  independence  exist  at  the  project  level.  The  Corps’  contractors, 
however,  have  a  process  for  reviewing  the  appropriate  information  related 
to  the  conflicts  of  interest  and  independence  of  the  experts  selected  for 
peer  review  panels  at  the  project  level. 

For  its  initial  peer  reviews,  the  Corps  relied  on  Battelle  to  establish  and 
manage  the  peer  review  panels.  From  2007  to  2009,  Battelle  managed  15 
independent  peer  reviews  for  the  Corps.  The  Corps  had  identified  Battelle 
as  a  potential  contractor  for  managing  its  peer  review  panels  as  early  as 
August  2007,  when  WRDA  2007  was  being  considered.  To  ensure  that 
Battelle  could  meet  the  section  2034  independence  requirements, 
according  to  Corps  officials,  the  Director  of  Civil  Works  and  the  Chief  of 
Planning  and  Policy  held  discussions  with  Battelle,  and  officials  from  the 
Corps’  Institute  for  Water  Resources  met  with  Battelle  to  discuss 
Battelle’s  existing  review  process  and  the  independent  peer  review 
requirements  of  WRDA  2007.  Battelle  informed  the  Corps  that  it  met  all 
WRDA  2007  requirements  for  an  eligible  organization,  and  Battelle 
identified  its  existing  contract  with  the  Army  Research  Office  as  a  vehicle 
for  employing  Battelle  to  establish  and  manaqe  peer  review  panels  under 
section  2034. 


During  that  time,  NAS  also  conducted  one  independent  peer  review  for 
the  Corps  on  the  Louisiana  Coastal  Protection  and  Restoration  Program, 
which  charged  the  Corps  with  developing  a  full  range  of  flood  control, 
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coastal  restoration,  and  hurricane  protection  measures  for  South 
Louisiana.  According  to  Corps  planning-center-of-expertise  officials, 
because  of  the  extensive  scope  and  breadth  of  the  project,  NAS  was 
chosen  instead  of  Battelle  to  conduct  that  peer  review.  But  Corps 
headquarters  and  planning-center-of-expertise  officials  told  us  that,  over 
the  course  of  NAS’  review,  they  realized  that  NAS  would  not  be  the 
appropriate  organization  for  reviewing  individual  projects  studies  because 
its  process  was  too  time-consuming  and  expensive.  A  member  of  that 
NAS  peer  review  panel  also  told  us  that  while  he  would  recommend  NAS 
review  for  larger  projects,  in  his  opinion  NAS  might  not  be  the  appropriate 
organization  for  reviewing  smaller  Corps  projects. 

In  2009,  the  Corps  sought  additional  contractors  to  establish  and  manage 
peer  review  panels  and  began  its  contractor  selection  process  by  putting 
out  a  request  for  proposals.  This  solicitation  included  as  contract 
requirements  the  section  2034  criteria  that  the  organizations  establishing 
and  managing  peer  review  panels  be  independent  and  free  from  conflicts 
of  interest.  The  Corps  received  six  proposals,  including  one  from  Battelle, 
and  each  of  these  proposals  was  then  evaluated  by  a  three-person 
review  panel.  The  panel  chairperson  told  us  that  the  section  2034  criteria 
that  eligible  organizations  be  independent  and  free  from  conflicts  of 
interest  were  considered  as  minimum  qualifications  for  screening  and 
selection.28  As  a  result  of  this  process,  the  Corps  awarded  a  contract  to 
Battelle — this  contract  was  in  addition  to  the  existing  contract  Battelle 
already  had  with  the  Corps  through  the  Army  Research  Office — and  one 
to  Noblis.29  The  Corps  determines  which  of  the  two  contractors  it  will  use 
to  manage  individual  peer  reviews  on  the  basis  of  the  contractors’ 
responses  to  specific  project  study  scope  of  work  requests,  described 
below. 

Although  the  Corps’  contractor  screening  and  selection  process  identifies 
general  contractor  independence  and  areas  of  conflicts  of  interest,  the 
Corps  does  not  have  a  process  for  reviewing  the  selected  contractors’ 


oo 

The  summary  evaluation  of  contract  proposals  does  not  discuss  the  section  2034 
eligibility  criteria;  although,  the  proposals  from  both  Battelle  and  Noblis  address  the  criteria 
and  state  that  those  organizations  meet  all  section  2034  criteria  for  eligible  organizations. 

oq 

Initially,  one  contract  was  awarded  to  Battelle  and  one  to  a  second  organization  whose 
contract  was  terminated  6  months  later  because  of  internal  management  issues  at  that 
organization.  In  November  2010,  on  the  basis  of  Noblis’s  proposal  from  the  original  panel 
evaluation,  the  Corps  selected  Noblis  to  replace  this  organization. 
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project-specific  independence  and  freedom  from  conflicts  of  interest.  For 
each  project  study  undergoing  a  peer  review,  the  Corps  sends  both 
contractors  a  “scope  of  work”  document,  which  describes  the  project 
study  and  lists  the  required  contractor  qualifications.  These  qualifications 
include  independence  and  freedom  from  conflicts  of  interest  related  to  the 
specific  project  study  being  reviewed.  In  response,  the  contractors  send 
the  Corps  their  proposals  for  conducting  the  peer  review,  which  generally 
include  statements  that  they  are  independent  and  free  from  project- 
specific  conflicts.30  Nevertheless,  we  identified  a  number  of  weaknesses 
in  the  Corps’  approach  for  reviewing  and  corroborating  this  information, 
including  the  following: 

•  The  Corps’  planning  centers  of  expertise  are  expected  to  review  the 
contractors’  overall  proposals,  but  the  Corps  does  not  require  the 
centers  to  ensure  that  contractors’  statements  of  independence  within 
the  proposals  are  reviewed  and  corroborated  for  each  individual 
project.  Although  planning-center-of-expertise  officials  told  us  that 
they  review  the  overall  proposals,  some  of  these  officials  also  stated 
that  they  did  not  believe  that  the  statements  required  review  because 
Corps  headquarters  had  already  prescreened  the  contractors  during 
the  initial  contractor  screening  and  selection  process.  Furthermore, 
the  Corps  has  not  provided  any  guidance  to  the  planning  centers  of 
expertise  or  other  Corps  offices  that  specifies  how  those  officials 
should  review  the  contractors’  project-specific  statements  at  the 
proposal  stage  and  ensure  that  they  are  accurate  and  that  the 
contractors  are  in  fact  independent  and  free  from  conflicts  of  interest. 
Absent  such  guidance,  the  Corps  cannot  ensure  that  its  contractors 
are  independent  and  free  from  conflicts  of  interest  at  the  project  level. 

•  The  Corps  neither  conducts  any  internal  conflicts-of-interest  checks 
nor  asks  contractors  for  documentation  about  potential  conflicts  of 
interest  so  that  it  can  determine  whether  a  conflict  exists;  rather,  the 
Corps  allows  the  contractors  to  make  that  determination  on  their  own. 
As  a  result,  if  the  contractors  do  not  provide  this  information  to  the 
Corps,  the  agency  does  not  have  a  process  for  otherwise  obtaining 
this  information. 


on 

Prior  to  the  selection  of  Noblis  as  a  contractor  in  201 0,  the  scope  of  work  went  solely  to 
Battelle.  Battelle  was  still  required  to  provide  the  Corps  with  a  proposal  describing  how  it 
would  conduct  the  peer  review  and  responding  to  the  contractor  requirements. 
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Unlike  the  Corps’  review  of  contractor  independence,  the  Corps’ 
contractors  do  solicit  and  review  information  on  panel  members’ 
independence  and  conflicts  of  interest  at  the  project  level.  The  contractors 
gather  information  about  prospective  panel  members  using  screening 
questions  developed  from  the  scope  of  work  for  each  peer  review.  These 
questions  cover  issues  described  in  the  NAS  policy  on  committee 
composition  and  conflicts  of  interest,  such  as  financial  and  employment 
interests  and  public  statements  and  positions.  The  peer  review  reports 
from  both  Battelle  and  Noblis  state  that  they  follow  both  the  OMB 
guidance  on  peer  review  and  the  NAS  policy  when  selecting  panel 
members.  According  to  contractors  and  Corps  officials,  district  and 
planning-center-of-expertise  officials  review  the  contractors’  screening 
questions,  as  well  as  the  resumes  of  selected  experts,  and  can  provide 
the  contractors  with  additional  information  about  potential  conflicts  of 
interest,  such  as  previous  work  a  particular  expert  may  have  done  for  the 
Corps.  Corps  officials  told  us  that  the  contractors  follow  up  on  such 
information  where  appropriate,  but  the  contractors  and  a  Corps  official  we 
spoke  to  said  that  it  is  the  contractors  who  ultimately  select  the  panel 
members  and  ensure  their  independence. 


The  Corps  Has 
Adopted  Most  Peer 
Review 

Recommendations, 
Resulting  in  Technical 
Improvements  but 
Generally  No  Changes 
in  Project  Decisions 


The  Corps  has  adopted  and  incorporated  most  of  the  peer  review 
recommendations  it  has  received.  Adoption  of  these  recommendations 
has  resulted  in  some  technical  improvements  to  project  study  reports  but 
generally  has  not  changed  the  Corps’  decisions  in  selecting  preferred 
project  designs.  According  to  some  Corps  officials  we  spoke  with,  this  is 
the  result  of  the  review  occurring  too  late  in  the  process  to  effect  a 
change  in  decision  making. 


The  Corps  Has  Adopted 
and  Incorporated  Most 
Peer  Review 
Recommendations, 
Resulting  in  Technical 
Improvements 


Of  the  49  project  studies  that  have  undergone  peer  review,  the  Corps  has 
provided  a  final  written  response  for  17.  The  Corps  has  adopted  231  of 
274  recommendations,  partially  adopted  31,  and  rejected  12  for  these  17 
peer  reviews.  Several  Corps  district  officials  told  us  that  they  make  every 
effort  not  to  reject  peer  review  recommendations  and  that  headquarters 
has  directed  them  to  adopt  recommendations  whenever  possible.  In  fact, 
some  district  officials  told  us  that  they  felt  pressure  from  headquarters  to 
adopt  peer  review  recommendations  even  when  the  recommendations 
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would  not  affect  the  study  outcome  and  would  be  burdensome  to 
implement. 

The  Corps’  adoption  of  peer  review  recommendations  has  improved  the 
technical  quality  of  its  project  study  reports,  according  to  Corps  officials 
and  panel  members  we  spoke  with.  Corps  officials  complimented  the 
quality  and  technical  competence  of  panel  members  and  stated  that  the 
panels’  recommendations  have  been  helpful  in  clarifying  and 
strengthening  the  arguments  presented  in  the  studies.  Most  of  the 
recommendations  either  requested  that  the  Corps  add  to  or  clarify  the 
study  report  or  stated  that  the  study  report  did  not  sufficiently  address 
certain  issues.  The  Corps  addressed  these  issues  in  almost  all  instances 
(193  of  201  recommendations)  within  its  written  responses  to  completed 
peer  review  reports.31 

A  smaller  number  of  recommendations  addressed  the  underlying 
assumptions  and  inputs  to  the  project  studies’  economic,  engineering, 
and  environmental  analyses.32  We  identified  four  instances  in  which  the 
Corps  revised  portions  of  its  analysis  on  the  basis  of  these  kinds  of 
recommendations.  In  none  of  these  cases  did  the  Corps  indicate  that  the 
revised  analyses  would  change  the  study  decisions.  In  one  case, 
according  to  Corps  documents,  the  revised  analysis  served  to  strengthen 
arguments  in  favor  of  its  recommended  plan.  In  response  to  a 
recommendation  concerning  an  environmental  analysis  from  the  peer 
review  of  the  Mid-Chesapeake  Bay  Islands  Ecosystem  Restoration 
project  study,  the  Corps  conducted  additional  analyses  to  justify  its 
calculations  of  environmental  benefits.  The  Corps  reported  that  the 
additional  analyses  led  to  the  determination  that  the  selected  plan  was 
appropriate  but  that  by  considering  the  ecosystem  impacts  of  the  project 
in  a  more  detailed  fashion,  justification  of  the  recommended  plan  was 
strengthened. 


O  1 

Forty-nine  studies  underwent  peer  review,  but  3  of  these  studies  resulted  in  2  peer 
review  reports,  so  the  total  number  of  reports  was  52.  For  the  52  completed  peer  review 
reports,  panel  members  made  a  total  of  910  recommendations.  Most  of  the 
recommendations  (672  of  910)  either  requested  that  the  Corps  add  to  or  clarify  the  study 
report  or  stated  that  the  study  report  did  not  sufficiently  address  certain  issues. 

op 

Specifically,  34  out  of  the  total  910  peer  review  recommendations  indicated  a  problem 
with  the  economic  analysis,  24  indicated  a  problem  with  the  engineering  analysis,  and  19 
indicated  a  problem  with  the  environmental  analysis. 
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Nevertheless,  despite  these  technical  improvements,  some  Corps  officials 
have  questioned  the  benefit  of  peer  review,  given  the  significant  amount 
of  time  that  district  staff  have  to  spend  managing  the  process  and 
responding  to  recommendations.  The  process  for  responding  to 
recommendations  begins  with  district  officials  drafting  a  written  response, 
which  they  provide  to  the  panel.  The  Corps’  response  to  the  peer  review 
recommendations  includes  a  detailed  description  of  the  steps  that  the 
Corps  has  taken  or  will  take  to  incorporate  the  recommendations  into  the 
project  study.  The  contractor  then  convenes  a  teleconference  at  which 
district  officials  discuss  the  draft  response  with  panel  members.  After  this 
discussion,  the  panel  members  provide  written  feedback — “backcheck 
responses” — to  the  Corps  stating  whether  they  agree  with  the  district’s 
response.33  The  district  then  finalizes  its  response  to  the 
recommendations  and  forwards  the  response  to  its  division  office.  After  its 
review,  the  division  forwards  the  response  to  headquarters,  where  the 
response  is  finalized.  The  final  written  response  is  generally  published  at 
the  same  time  as  the  final  decision  document  for  the  project  study.  The 
time  between  completion  of  the  peer  review  report  and  public  availability 
of  Corps’  written  response  therefore  varies  greatly  depending  on  the 
individual  project.  In  one  case  it  was  3  months,  while  in  other  cases  peer 
review  reports  have  been  completed  for  more  than  3  years  without  a  final 
response  from  the  Corps  having  been  made  public. 


For  Most  Studies,  Peer 
Review  Occurs  Too  Late  in 
the  Study  Process  to  Affect 
Decision  Making 


Corps  officials  we  spoke  with  told  us  that  peer  review  recommendations 
have  generally  had  no  impact  on  the  Corps’  decision  making  process. 
These  Corps  officials  were  not  aware  of  any  project  studies  for  which  the 
study  outcome  changed  as  a  result  of  peer  review.  Corps  headquarters 
officials  told  us  that  one  reason  for  the  lack  of  impact  of  peer  review  on 
decision-making  is  because  the  Corps’  internal  review  process  is 
identifying  the  same  issues  as  peer  review.  Another  reason  cited  by 
Corps  officials  for  the  lack  of  impact  on  decision  making  is  the  fact  that 
peer  review  is  occurring  at  the  end  of  the  study  process.  Peer  review 
generally  occurs  concurrently  with  the  public  comment  period  for  the  draft 
study  report,  which  comes  after  the  preferred  design  has  been  selected. 
As  a  result,  some  recommendations  about  alternatives  may  not  have 


oo 

The  backcheck  responses  are  not  part  of  the  panel’s  final  peer  review  report  which 
contains  only  the  panel’s  recommendations.  The  backcheck  responses  are  given  after  the 
final  peer  review  report  is  submitted  to  ensure  that  the  panel’s  opinion  and  objectivity  are 
not  influenced  by  the  Corps. 
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been  implemented  because  the  decision  on  the  preferred  design  had 
already  been  made.  Selecting  a  different  preferred  design  at  that  stage 
would  require  the  Corps  to  revisit  an  already  completed  selection  analysis 
and  decision.  For  example,  in  the  peer  review  report  on  the  Cedar  River- 
Cedar  Rapids  Iowa  Flood  Risk  Management  project  study,  the  review 
panel  recommended  that  the  Corps  further  investigate  one  of  the  non- 
selected  design  alternatives,  because  panel  members  felt  that  the 
alternative  might  achieve  project  objectives  better  than  the  preferred 
design.  The  Corps,  however,  had  already  selected  its  design  and  decided 
to  proceed.  The  Corps  did  not  adopt  this  recommendation,  stating  that  it 
believed  its  analysis  of  alternatives  was  sound  and  that  there  was  no 
reasonable  expectation  that  a  more  detailed  analysis  of  the  alternative 
would  result  in  finding  that  it  had  greater  net  economic  benefits  than  the 
preferred  design. 

In  contrast,  when  the  Corps  has  conducted  a  peer  review  earlier  in  the 
process,  opportunities  have  arisen  for  positive  impacts  on  a  study 
decision.  For  example,  the  American  River  Common  Features  project 
study  peer  review  was  conducted  early  in  the  study  process.  According  to 
Corps  division  and  planning-center-of-expertise  officials,  they  conducted 
peer  review  early  to  obtain  external  input  on  defining  the  problem  and  to 
inform  decision-making  due  to  the  complexity  of  the  project.  As  a  result, 
the  peer  review  began  before  the  alternative  formulation  briefing,  when 
the  without-project  conditions  were  being  identified.  By  employing  this 
approach,  the  Corps  received  feedback  from  the  review  panel  before 
selecting  the  preferred  design.  The  panel’s  recommendations  included 
three  suggested  changes  to  the  Corps’  analyses  and  model  calibrations, 
which  the  Corps  had  time  to  incorporate  before  conducting  the  alternative 
analysis  and  selection.  According  to  the  contractor  that  managed  the  peer 
review,  the  panel  members  involved  in  the  American  River  Common 
Features  peer  review  also  found  the  timing  of  the  review  to  be  beneficial 
and  suggested  that  the  Corps  conduct  peer  review  earlier  for  other  project 
studies. 

The  timing  of  peer  review  was  also  addressed  in  the  2002  NAS  study  on 
peer  review.  NAS  recommended  that  the  Corps  initiate  peer  reviews  early 
enough  in  the  study  process  so  that  the  review  results  could  be 
meaningfully  incorporated  into  the  study  or  project  design  and  stated  that 
conducting  peer  review  before  selecting  a  recommended  plan  is  essential 
if  the  Corps  is  to  benefit  from  the  review.  Corps  officials  nevertheless  told 
us  that  they  have  generally  chosen  to  conduct  peer  reviews  later  in  the 
process  to  minimize  effects  on  project  study  schedules.  Corps 
headquarters  officials  noted  that,  for  many  studies,  peer  review  occurred 


Page  31 


GAO-12-352  Army  Corps  of  Engineers 


late  in  the  process  because  the  studies  were  under  way  at  the  time  the 
Corps  began  requiring  peer  review.  These  officials  also  noted  that  it 
would  be  challenging  to  assemble  a  peer  review  panel  to  conduct  a 
review  early  in  the  study  process  and  retain  the  same  panel  to  complete 
this  review  at  the  end  of  the  study.  Furthermore,  Corps  headquarters 
officials  noted  that  a  further  challenge  is  implementing  section  2033 
WRDA  requirements  along  with  section  2034.  Section  2033  generally 
requires  the  Corps  to  complete  feasibility  studies  within  2  years. 
According  to  Corps  officials,  there  is  tension  between  these  requirements 
and  it  may  be  challenging  to  include  peer  review  throughout  the  study 
process  without  altering  project  study  schedules. 


Conclusions 


Section  2034  established  a  trial  to  look  at  the  cost  and  impact  of 
conducting  peer  review  for  controversial  and  costly  projects  over  a  7-year 
period.  After  the  trial  period,  based  on  information  provided  by  the  Corps, 
Congress  could  reconsider  whether  to  retain  or  revise  section  2034  or 
allow  it  to  lapse.  Because  the  Corps  generally  does  not  specify  the 
authority  under  which  peer  review  was  conducted,  however,  it  has  not 
provided  Congress  with  the  information  needed  to  evaluate  the  merits  of 
the  section  2034  requirements.  In  addition,  the  Corps’  implementation  of 
peer  review  has  not  focused  on  the  larger,  more  complex,  and 
controversial  projects  that  were  contemplated  when  section  2034  was 
enacted  and  as  recommended  by  NAS  a  decade  ago.  As  a  result,  project 
studies  are  being  selected  to  undergo  peer  review  that  may  not  be 
warranted  and  may  thereby  be  increasing  project  costs  and  schedules 
needlessly.  Further,  essential  to  the  integrity  of  the  peer  review  process  is 
the  assurance  that  the  Corps  has  effective  processes  not  only  to  ensure 
overall  contractor  independence  and  freedom  from  conflicts  of  interest  but 
also  to  ensure  project-level  independence  and  freedom  from  conflicts  of 
interest.  The  Corps’  current  process,  however,  has  a  number  of 
weaknesses  with  respect  to  ensuring  no  conflicts  of  interest  exist  at  the 
project  level.  Finally,  with  peer  review  generally  occurring  late  in  the 
Corps’  project  study  process,  peer  review  serves  more  to  strengthen  the 
Corps’  presentation  of  its  decisions  than  to  influence  its  decision  making. 
This  effect  runs  counter  to  what  NAS  recommended  in  2002,  that 
realizing  the  benefits  of  peer  review  requires  the  results  to  be  used  as 
inputs  in  the  decision-making  process.  By  choosing  to  apply  peer  review 
late  in  the  project  study  process,  the  Corps  has  effectively  chosen  to  not 
use  the  results  of  peer  review  to  enhance  its  decision-making  process 
and  ensure  selection  of  the  most  effective  project  alternatives. 
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Recommendations  for  recommenc'  that  the  Secretary  of  Defense  direct  the  Chief  of 

Engineers  and  Commanding  General  of  the  U.S.  Army  Corps  of 
Executive  Action  Engineers  to  take  the  following  three  actions: 

To  facilitate  congressional  evaluation  of  the  7-year  trial  period  outlined  in 
section  2034,  the  Corps  should: 

•  Identify  for  each  past  and  future  peer  review  the  specific  statutory 
authority  under  which  the  peer  review  was  conducted  and  the  criteria 
triggering  peer  review  under  the  Corps’  civil  works  review  policy. 

To  better  reflect  section  2034  and  provide  more  effective  stewardship  of 
public  resources  and  ensure  efficient  and  effective  operations,  the  Corps 
should: 

•  Revise  the  criteria  in  the  Corps’  process  for  conducting  peer  review  to 
focus  on  larger,  more  complex,  and  controversial  projects;  to 
encourage  peer  review  to  occur  earlier  in  the  study  process;  and  to 
include  exclusions  to  peer  review  that  align  with  section  2034. 

•  Develop  a  documented  process  to  ensure  that  contractors  are 
independent  and  free  from  conflicts  of  interest  on  a  project-specific 
basis. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  the  Department  of  Defense  for  review 
and  comment.  In  its  written  comments,  reprinted  in  appendix  III,  the 
department  generally  concurred  with  our  recommendations.  Specifically, 
in  response  to  our  first  recommendation,  the  department  agreed  that  the 
Corps  should,  and  stated  that  it  will,  identify  for  each  past  and  future  peer 
review  the  specific  statutory  authority  under  which  the  peer  review  was 
conducted  and  the  criteria  triggering  peer  review  under  the  Corps’  civil 
works  review  policy.  In  response  to  our  second  recommendation,  the 
department  partially  concurred,  stating  that  it  agreed  that  peer  review 
should  be  focused  on  studies  that  will  significantly  benefit  from  peer 
review  and  that  initiating  reviews  early  is  advantageous.  Nevertheless, 
the  department  noted  that  early  involvement  must  be  balanced  with 
having  sufficient  data  and  analysis  available  for  review  and  also 
highlighted  work  under  way  at  the  agency  to  overhaul  its  planning 
processes,  which  includes  efforts  to  better  align  product  reviews  for 
greater  effectiveness.  In  response  to  our  third  recommendation,  the 
department  agreed  that  the  Corps  should  develop  a  documented  process 
to  ensure  that  contractors  are  independent  and  free  from  conflicts  of 
interest  on  a  project-specific  basis. 
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Although  the  department  generally  concurred  with  our  recommendations, 
it  disagreed  with  our  report’s  finding  that  the  Corps’  process  does  not  use 
the  flexibility  provided  in  section  2034,  and  it  disagreed  that  some  studies 
have  undergone  review  that  did  not  warrant  it.  The  department  stated  that 
the  Corps  has  carefully  deliberated  in  support  of  the  agency  decision  to 
conduct  peer  review  on  the  three  studies  noted  in  our  report  and  also 
stated  that  the  Corps  stands  by  all  of  its  decisions  to  date  to  grant  or  deny 
exclusions  from  peer  review.  Nevertheless,  the  department  stated  that  as 
part  of  the  Corps’  ongoing  review  of  the  civil  works  review  policy,  it  will 
assess  the  effectiveness  of  its  criteria  and  how  the  criteria  are  applied  to 
determine  which  studies  should  be  considered  for  exclusion.  In  addition, 
the  department  expressed  concern  about  the  level  of  weight  given  in  the 
report  to  anecdotal  remarks  from  field-level  officials,  who  in  the 
department’s  opinion  may  not  have  had  the  benefit  of  the  corporate  vision 
supporting  the  Army  Civil  Works  Program.  We  disagree  with  the 
department’s  characterization  of  our  methodology.  As  clearly  described  in 
the  scope  and  methodology  section  of  this  report,  we  interviewed  officials 
who  had  a  corporate-level  perspective,  as  well  as  those  who  had  a 
project-level  perspective.  Specifically,  to  obtain  a  corporate-level  view,  we 
interviewed  senior  level  officials  from  Corps  headquarters,  the  Institute  for 
Water  Resources,  and  the  planning  centers  of  expertise  involved  in 
managing  the  peer  reviews.  In  addition,  to  get  a  project-level  perspective 
and  to  assess  the  impact  of  peer  review  on  division  and  district  offices, 
we  interviewed  officials  in  all  of  the  Corps’  eight  divisions,  and  from  10 
geographically  dispersed  Corps  districts  that  had  conducted  studies  that 
underwent  peer  review.  We  also  interviewed  the  three  contractors  and 
selected  peer  review  panel  members  and  local  sponsors  of  Corps  civil 
works  projects.  We  believe  that  the  report  provides  a  balanced 
perspective  from  both  the  headquarters  and  field  levels. 


As  agreed  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  of  this  report  to  the 
appropriate  congressional  committees,  the  Secretary  of  Defense,  the 
Chief  of  Engineers  and  Commanding  General  of  the  U.S.  Army  Corps  of 
Engineers,  and  other  interested  parties.  This  report  will  be  available  at  no 
charge  on  the  GAO  website  at  http://www.gao.gov. 
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If  you  or  your  staff  members  have  any  questions  about  this  report,  please 
contact  me  at  (202)  512-3841  or  mittala@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to  this 
report  are  listed  in  appendix  IV. 

Sincerely  yours, 


Anu  K.  Mittal 

Director,  Natural  Resources  and  Environment 


Page  35 


GAO-12-352  Army  Corps  of  Engineers 


Appendix  I:  Objectives,  Scope,  and 
Methodology 


Our  objectives  for  this  work  were  to  examine  (1 )  the  number  of  Corps 
project  studies  that  have  undergone  independent  peer  review  in  response 
to  section  2034  of  the  Water  Resources  and  Development  Act  (WRDA)  of 
2007,  (2)  the  cost  of  these  peer  reviews,  (3)  the  extent  to  which  the  U.S. 
Army  Corps  of  Engineers’  (Corps)  process  for  determining  if  a  project 
study  is  subject  to  peer  review  is  consistent  with  section  2034,  (4)  the 
process  the  Corps  uses  to  ensure  that  the  contractors  it  hires  and  the 
experts  the  contractors  select  to  review  project  studies  are  independent 
and  free  from  conflicts  of  interest,  and  (5)  the  extent  to  which 
recommendations  from  peer  reviews  have  been  incorporated  into  project 
studies.  We  focused  on  peer  reviews  for  which  reports  had  been 
completed  since  WRDA  2007  was  enacted. 

To  address  all  of  these  objectives,  we  reviewed  relevant  legal 
requirements,  policy  guidance,  review  plans,  and  peer  review  reports  for 
project  studies  that  were  subject  to  a  peer  review  and  for  which  a  peer 
review  report  had  been  completed  since  WRDA  2007  was  enacted.  In 
addition,  we  selected  a  nongeneralizable  sample  of  six  peer  reviews  to 
examine  in  greater  depth  to  better  understand  the  costs  associated  with 
conducting  these  reviews,  as  well  as  the  overall  impact  of  the  process  on 
the  timeline  of  the  project  study  and  the  study  outcome.  We  chose  these 
reviews  as  illustrative  examples  and  selected  one  from  each  of  the  Corps’ 
planning  centers  of  expertise  and  at  least  one  for  each  of  the  three 
contractors  the  Corps  has  used  to  manage  peer  reviews  since  enactment 
of  WRDA  2007.  Although  the  information  derived  from  analysis  of  these 
case  studies  cannot  be  generalized,  these  examples  provide  valuable 
insights  into  the  peer  review  process.  We  conducted  semistructured 
interviews  with  officials  from  Corps  headquarters,  the  planning  centers  of 
expertise  involved  in  managing  the  peer  reviews,  all  of  the  Corps’  eight 
divisions,  and  from  10  geographically  dispersed  Corps  districts  that  had 
conducted  studies  that  underwent  peer  review.  We  also  conducted 
semistructured  interviews  with  the  three  contractors,  as  well  as  selected 
peer  review  panel  members  and  local  sponsors  of  Corps  civil  works 
projects. 

To  determine  the  number  of  studies  that  have  undergone  peer  review  in 
response  to  section  2034  of  WRDA,  we  reviewed  all  completed  peer 
review  reports,  plus  Corps  reports  and  information  on  completed  peer 
reviews.  We  reviewed  information  on  completed  peer  reviews  obtained 
from  headquarters,  the  planning  centers  of  expertise,  divisions,  and 
selected  districts.  We  also  reviewed  information  on  completed  peer 
reviews  obtained  from  the  contractors  that  established  the  peer  review 
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panels  and  the  entities  the  Corps  used  to  administer  these  contracts:  the 
Institute  for  Water  Resources  and  the  Army  Research  Office. 

To  determine  the  cost  of  these  reviews,  we  reviewed  contract  award 
documents  and  information  on  contract  costs  from  the  contractors. 
Generally,  we  relied  on  the  contract  award  amounts  reported  in  the 
contracts  to  determine  the  cost  of  the  contracts  awarded  for  establishing 
review  panels.  For  four  contract  awards,  the  contract  work  included 
establishing  a  peer  review  panel  and  additional  work.  For  these  awards, 
we  therefore  relied  on  information  provided  by  the  contractor  on  the 
portion  of  the  contract  cost  that  was  for  the  peer  review.  For  the  contract 
award  for  peer  review  of  a  local  sponsor-led  study,  we  relied  on 
information  from  the  local  sponsor  and  the  contractor  on  the  cost  of  the 
award.  In  addition,  for  the  six  case  study  peer  reviews,  we  analyzed 
information  on  costs  associated  with  managing  the  review  process, 
including  cost  data  and  estimates  provided  by  districts  with  regard  to 
district  and  other  staff  time  involved  in  peer  review.  In  cases  where  we 
reported  cost  data  including  staff  time  associated  with  completing  peer 
review,  we  asked  knowledgeable  officials  about  the  data  system  and  the 
quality  of  the  data  and  determined  that  they  were  sufficiently  reliable  for 
our  purposes.  In  cases  where  we  reported  estimates  of  these  costs,  we 
asked  officials  about  how  these  estimates  were  developed  and 
determined  that  they  were  sufficiently  reliable  for  our  purposes. 

To  determine  the  extent  to  which  the  Corps’  process  for  determining  if  a 
study  is  subject  to  peer  review  is  consistent  with  section  2034,  we 
analyzed  the  legal  requirements  and  relevant  policy  guidance  for 
determining  when  to  conduct  peer  review.  We  also  reviewed 
documentation  on  decisions  to  conduct  peer  review  included  in  review 
plans  and  documents  requesting  exclusion  from  peer  review.  In  addition, 
we  reviewed  information  on  the  characteristics  of  studies  that  underwent 
peer  review,  including  date  initiated,  whether  an  environmental  impact 
statement  was  included,  and  the  type  of  study.  We  identified  this 
information  in  review  plans,  study  drafts,  signed  Chiefs  reports,  and  other 
Corps  study-related  documents;  Corps  officials  from  relevant  districts  or 
divisions  reviewed  this  information. 

To  determine  the  process  the  Corps  uses  to  ensure  that  the  contractors  it 
hires  and  the  experts  the  contractors  select  are  independent  and  free 
from  conflicts  of  interest,  we  reviewed  information  on  contractor  selection 
obtained  from  Corps  headquarters  and  the  Institute  for  Water  Resources. 
We  also  reviewed  documentation  from  the  contractors  that  outlined 
contractor  and  reviewer  qualifications,  as  well  as  the  National  Academy  of 
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Sciences  Policy  on  Committee  Composition  and  Balance  and  Conflicts  of 
Interest. 

To  determine  the  extent  to  which  peer  review  recommendations  are 
incorporated  into  project  studies,  we  reviewed  information  obtained  from 
headquarters,  the  planning  centers  of  expertise,  divisions,  and  selected 
districts  on  how  the  Corps  responds  to  peer  review  recommendations. 

We  also  reviewed  all  peer  review  recommendations  contained  in 
completed  peer  review  reports,  as  well  as  all  responses  to  peer  review 
recommendations  contained  in  the  Corps’  published  responses  to  the 
completed  peer  review  reports. 

We  conducted  this  performance  audit  from  April  201 1  to  March  2012  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Page  38 


GAO-12-352  Army  Corps  of  Engineers 


Appendix  II:  Characteristics  of  Studies  That 
Underwent  Peer  Review 


Forty-six  of  the  49  peer  reviews  completed  in  table  2  below  were 
conducted  by  Battelle  Memorial  Institute;  Noblis  completed  the  Green  Bay 
Dredged  Material  Management  Plan  and  the  Wood  River  Levee  System 
General  Reevaluation  Report  peer  reviews,  and  the  National  Academy  of 
Sciences  conducted  the  Louisiana  Coastal  Protection  and  Restoration 
peer  review.  As  table  2  shows,  the  studies  that  underwent  peer  review 
came  under  the  areas  of  ecosystem  restoration  (19  of  49),  flood  risk 
management  (15  of  49),  deep  draft  navigation  (7  of  49),  coastal  storm 
damage  reduction  (5  of  49),  inland  navigation  (2  of  49),  and  water 
management  and  reallocation  (1  of  49).  According  to  our  analysis  of 
Corps  documents,  32  of  the  49  studies  included  an  environmental  impact 
statement  (EIS),  19  of  49  were  initiated  after  November  2005,  and  42  of 
49  had  an  estimated  total  project  cost  greater  than  $45  million. 


Table  2:  Characteristics  of  Studies  That  Underwent  Peer  Review 


Dollars  in  thousands 


Study  name 
and  type 

Corps  district 

Corps  division 

Corps 
planning 
center  of 
expertise 

Date  of 
peer 
review 
report(s) 

Feasibility 

or 

reevaluation 

study3 

Initiated 

after 

Nov. 

2005 

Estimated 

cost 

exceeding 
$45  million 

EIS 

Cost  of 
peer 
review 
contract 

Alton  to  Gale 

Organized  Levee 

Districts 

(Continuing, 

Deficiency 

Corrections)  Letter 

Report 

St.  Louis 

Mississippi 

Valley 

Flood  risk 
management 

Aug.  18, 
2010 

V 

$145 

American  River 
Draft  Natomas 
Post-Authorization 
Change  Report 
and  Draft  EISb 

Sacramento 

South 

Pacific 

Flood  risk 
management 

June  10, 

2009  & 
Sept.  16, 

2010 

V 

V 

V 

V 

458 

Barataria  Basin 
Barrier  Shoreline 
Restoration  Draft 
Construction 

Report  and  Draft 
EIS 

New  Orleans 

Mississippi 

Valley 

Ecosystem 

restoration 

Oct.  3, 
2011 

V 

V 

V 

141 

Biscayne  Bay 
Coastal  Wetlands 
Project 

Implementation 

Report 

Jacksonville 

South  Atlantic 

Ecosystem 

restoration 

Dec.  1, 
2009 

V 

V 

V 

164 

Page  39 


GAO-12-352  Army  Corps  of  Engineers 


Appendix  II:  Characteristics  of  Studies  That 

Underwent  Peer  Review 

Dollars  in  thousands 

Study  name 
and  type 

Corps  district 

Corps  division 

Corps 
planning 
center  of 
expertise 

Date  of 
peer 
review 
report(s) 

Feasibility 

or 

reevaluation 

study3 

Initiated 

after 

Nov. 

2005 

Estimated 

cost 

exceeding 
$45  million 

EIS 

Cost  of 
peer 
review 
contract 

Boston  Harbor 
Navigation 
Improvement  Draft 
Feasibility  Study 
and  Draft 
Supplemental  EIS 

New  England 

North  Atlantic 

Deep  draft 
navigation 

June  3, 
2008 

V 

V 

V 

159 

Brevard  County 

Mid-Reach 

Shoreline 

Protection  Project 
Draft  Integrated 
General  Re- 
evaluation  Report 
and  Supplemental 
EIS 

Jacksonville 

South  Atlantic 

Coastal  storm 

damage 

reduction 

Dec.  9, 
2009 

V 

V 

V 

173 

Calcasieu  River 
and  Pass  Dredged 
Material 

Management  Plan 
and  Supplemental 
EIS 

New  Orleans 

Mississippi 

Valley 

Deep  draft 
navigation 

Aug.  29, 
2008 

V 

V 

135 

Cedar  Rapids 

Flood  Risk 
Management 
Feasibility  Study 
with  Integrated 
Environmental 
Assessment 

Rock  Island 

Mississippi 

Valley 

Flood  risk 
management 

Oct.  28, 
2010 

V 

V 

V 

122 

Chacon  Creek, 

Rio  Grande  Draft 
Feasibility  Report 
and  Integrated 
Environmental 
Assessment 

Fort  Worth 

Southwestern 

Flood  Risk 
management 

Nov.  17, 
2010 

V 

137 

Chatfield  Storage 
Reallocation  Study 
and  EIS 

Omaha 

Northwestern 

Water 

management/ 

reallocation 

Oct.  25, 
2011 

V 

V 

134 

Clear  Creek  Risk 
Management 
General 
Reevaluation 

Report  and 
Preliminary  Draft 
EISb,c 

Galveston 

Southwestern 

Flood  risk 
management 

May,  12, 
2009 

V 

V 

V 

290 
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Dollars  in  thousands 

Study  name 
and  type 

Corps  district 

Corps  division 

Corps 
planning 
center  of 
expertise 

Date  of 
peer 
review 
report(s) 

Feasibility 

or 

reevaluation 

study3 

Initiated 

after 

Nov. 

2005 

Estimated 

cost 

exceeding 
$45  million 

EIS 

Cost  of 
peer 
review 
contract 

Columbia  River  at 
the  Mouth  Major 
Rehabilitation 
Evaluation  Report 

Portland 

Northwestern 

Deep  draft 
navigation 

Mar.  9, 
2011 

V 

200 

East  Branch  Dam, 
Clarion  River,  Elk 
County,  Dam 

Safety 

Modification 

Report 

Pittsburgh 

Great  Lakes 
and  Ohio  River 

Flood  risk 
management 

July  2, 
2010 

V 

V 

105 

East  St.  Louis 

Flood  Protection 

Limited 

Reevaluation 

Report  and 
Environmental 
Assessment  on 
Design  Deficiency 
Corrections 

St.  Louis 

Mississippi 

Valley 

Flood  risk 
management 

Aug.  3, 
2010 

V 

V 

V 

159 

Environmental 

DNA  (eDNA) 
Science  and 
Methodology 

N/Ad 

Great  Lakes 
and  Ohio  River 

Ecosystem 

restoration 

Dec.  7, 
2010 

V 

151 

Fargo-Moorhead 
Flood  Risk 
Management 
Feasibility  Studyb  e 

St.  Paul 

Mississippi 

Valley 

Flood  risk 
management 

May  17, 
2010  & 
July  7, 
2011 

V 

V 

V 

V 

282 

Freeport  Harbor 
Draft  Feasibility 
Report  and  EISe 

Galveston 

Southwestern 

Deep  draft 
navigation 

Aug.  20, 
2008 

V 

V 

V 

214 

Green  Bay 

Dredged  Material 
Management  Plan 

Detroit 

Great  Lakes 
and  Ohio  River 

Inland 

navigation 

June  27, 
2011 

V 

V 

101 

Jamaica  Bay, 
Marine  Park,  and 
Plumb  Beach 

Draft  Interim 
Feasibility  Report 

New  York 

North  Atlantic 

Ecosystem 

restoration 

Dec.  8, 
2010 

V 

V 

164 

Kissimmee  River 
Restoration  Post- 
Authorization 
Change  Limited 
Reevaluation 

Report 

Jacksonville 

South  Atlantic 

Ecosystem 

restoration 

Oct.  13, 
2010 

V 

V 

V 

98 

Page  41 


GAO-12-352  Army  Corps  of  Engineers 


Appendix  II:  Characteristics  of  Studies  That 
Underwent  Peer  Review 


Dollars  in  thousands 

Corps 

Date  of 

Feasibility 

Initiated 

Estimated 

Cost  of 

planning 

peer 

or 

after 

cost 

peer 

Study  name 

center  of 

review 

reevaluation 

Nov. 

exceeding 

review 

and  type 

Corps  district 

Corps  division 

expertise 

report(s) 

study3 

2005 

$45  million 

EIS 

contract 

L-31N  Seepage 

Jacksonville 

South  Atlantic 

Ecosystem 

Mar.  10, 

102 

Management  Pilot 
Project  Draft 
Integrated  Pilot 
Project  Design 
Report  and 
Environmental 

restoration 

2009 

Assessment 

Louisiana  Coastal 

New  Orleans 

Mississippi 

Ecosystem 

June  23, 

V 

V 

V 

srF 

Area  Amite  River 
Diversion  Canal 
Modification 

Valley 

restoration 

2010 

Integrated 
Feasibility  Study 
and  Supplemental 
EIS 

Louisiana  Coastal 

New  Orleans 

Mississippi 

Ecosystem 

June  25, 

V 

V 

V 

V 

srF 

Area  Convey 
Atchafalaya  River 
Water  to  Northern 
Terrebonne 

Valley 

restoration 

2010 

Marshes, 

Lafourche 

Terrebonne,  St. 
Mary  Parish, 
Integrated 
Feasibility  Study 
and  EIS 

Louisiana  Coastal 

New  Orleans 

Mississippi 

Ecosystem 

June  23, 

V 

V 

V 

V 

srF 

Area  Medium 
Diversion  at  White 
Ditch, 

Plaquemines 

Parish,  Integrated 
Feasibility  Study 
and  Supplemental 
EIS 

Valley 

restoration 

2010 

Louisiana  Coastal 

New  Orleans 

Mississippi 

Ecosystem 

June  22, 

V 

V 

V 

V 

srF 

Area  Small 
Diversion  at 
Convent/Blind 

Valley 

restoration 

2010 

River,  St.  James 
Parish,  Integrated 
Feasibility  Study 
and  EIS 
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Dollars  in  thousands 

Corps 

Date  of 

Feasibility 

Initiated 

Estimated 

Cost  of 

planning 

peer 

or 

after 

cost 

peer 

Study  name 

center  of 

review 

reevaluation 

Nov. 

exceeding 

review 

and  type 

Corps  district 

Corps  division 

expertise 

report(s) 

study3 

2005 

$45  million 

EIS 

contract 

Louisiana  Coastal 

New  Orleans 

Mississippi 

Ecosystem 

June  25, 

V 

V 

V 

stF 

Area  Terrebonne 
Basin  Barrier 
Shoreline 
Restoration, 
Terrebonne 

Valley 

restoration 

2010 

Parish,  Integrated 
Feasibility  Study 
and  EIS 

Louisiana  Coastal 

New  Orleans 

Mississippi 

Coastal  storm 

2009 

V 

N/A 

V 

595 

Protection  and 

Valley 

damage 

Restoration 

Program  Draft 

Final  Technical 
Report9 

reduction 

Marlinton  Detailed 

Huntington 

Great  Lakes 

Flood  risk 

Nov.  16, 

V 

V 

V 

191 

Project  Report  and 

and  Ohio  River 

management 

2010 

EIS 


Melvin  Price  Wood 
River 

Underseepage 
Limited 
Reevaluation 
Report  and 
Environmental 
Assessment  on 
Design  Deficiency 
Corrections 

St.  Louis 

Mississippi 

Valley 

Flood  risk 
management 

Apr.  6, 

2011 

V 

V 

130 

Middle 

Chesapeake  Bay 
Islands  Ecosystem 
Restoration  Final 
Integrated 
Feasibility  Report 
and  EIS  and 
supporting 
documentation 

Baltimore 

North  Atlantic 

Ecosystem 

restoration 

Jan.  23, 
2008 

V 

V 

V 

121 

Mississippi 

Coastal 

Improvements 

Program 

Comprehensive 

Plan 

Mobile 

South  Atlantic 

Coastal  storm 

damage 

reduction 

Nov.  7, 

2008 

V 

V 

V 

161 
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Dollars  in  thousands 

Study  name 
and  type 

Corps  district 

Corps  division 

Corps 
planning 
center  of 
expertise 

Date  of 
peer 
review 
report(s) 

Feasibility 

or 

reevaluation 

study3 

Initiated 

after 

Nov. 

2005 

Estimated 

cost 

exceeding 
$45  million 

EIS 

Cost  of 
peer 
review 
contract 

Mississippi  River 
Gulf  Outlet 
Ecosystem 
Restoration  Plan 
Feasibility  Study 
and  EIS 

New  Orleans 

Mississippi 

Valley 

Ecosystem 

restoration 

June  3, 
2011 

V 

V 

V 

V 

373 

Mohawk  Dam 

Major 

Rehabilitation 

Report 

Huntington 

Great  Lakes 
and  Ohio  River 

Flood  risk 
management 

Apr.  22, 
2011 

V 

V 

240 

Navigation  and 
Ecosystem 
Sustainability 
Program,  Project 
P2,  Lock  and  Dam 
22  Fish  Passage 
Improvement 
Project 

Implementation 

Report 

Rock  Island 

Mississippi 

Valley 

Ecosystem 

restoration 

Dec.  17, 
2009 

V 

V 

141 

Olmsted  Post- 
Authorization 
Change  Reportbe 

Louisville 

Great  Lakes 
and  Ohio  River 

Inland 

navigation 

Nov.  15, 
2010  & 
Nov.  4, 
2011 

V 

V 

V 

201 

Port  Everglades 
Harbor  Science 
Reports  for  the 
Feasibility  Study 
and  EIS 

Jacksonville 

South  Atlantic 

Ecosystem 

restoration 

Aug.  17, 
2011 

V 

V 

V 

76 

Programmatic  EIS 
for  the  Mechanical 
Creation  and 
Maintenance  of 
Emergent 

Sandbar  Habitat 
on  the  Upper 
Missouri  River 

Omaha 

Northwestern 

Ecosystem 

restoration 

Mar.  17, 
2010 

V 

V 

120 

Sabine  Neches 
Waterway 

Channel 

Improvement  Plan 
Draft  Feasibility 
Report,  Draft  EIS, 
and  Supporting 
Documentation6 

Galveston 

Southwestern 

Deep  draft 
navigation 

Dec.  13, 
2007 

V 

V 

V 

313 
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Dollars  in  thousands 

Study  name 
and  type 

Corps  district 

Corps  division 

Corps 
planning 
center  of 
expertise 

Date  of 
peer 
review 
report(s) 

Feasibility 

or 

reevaluation 

study3 

Initiated 

after 

Nov. 

2005 

Estimated 

cost 

exceeding 
$45  million 

EIS 

Cost  of 
peer 
review 
contract 

Sacramento  River 
Deep  Water  Ship 
Channel  Limited 
Reevaluation 

Study  and 
Supplemental  EIS 

San 

Francisco 

South  Pacific 

Deep  draft 
navigation 

Sep.  30, 
2011 

V 

V 

V 

164 

San  Clemente 
Storm  Damage 
and  Shoreline 
Protection 
Feasibility  Study 

Los  Angeles 

South  Pacific 

Coastal  storm 

damage 

reduction 

July  23, 
2010 

V 

V 

145 

Savannah  Harbor 
Expansion  Project 
General 
Reevaluation 
Report0 

Savannah 

South  Atlantic 

Deep  draft 
navigation 

Feb.  11, 
2011 

V 

V 

V 

484 

St.  Johns  Bayou 
and  New  Madrid 
Floodway 
Consolidated 

NEPA  Document 
and  Work  Plan 
(Phase  l)h 

Memphis 

Mississippi 

Valley 

Flood  risk 
management 

Oct.  23, 
2009 

V 

V 

453 

Success  Dam 
Remediation  Dam 
Safety  Assurance 
Program  Letter 
Report 

Sacramento 

South  Pacific 

Flood  risk 
management 

Mar.  24, 
2011 

V 

V 

285 

Surf  City  and 

North  Topsail 

Beach  Draft 
Integrated 
Feasibility  Report 
and  EIS 

Wilmington 

South  Atlantic 

Coastal  storm 

damage 

reduction 

Apr.  16, 
2010 

V 

V 

V 

187 

Tamiami  Trail 

Limited 

Reevaluation 

Report 

Jacksonville 

South  Atlantic 

Ecosystem 

restoration 

June  2, 
2008 

V 

V 

142 

Tres  Rios  del 

Norte,  Pima 

County,  Feasibility 
Study 

Los  Angeles 

South  Pacific 

Ecosystem 

restoration 

Sep.  26, 
2011 

V 

V 

V 

159 
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Dollars  in  thousands 

Study  name 
and  type 

Corps  district 

Corps  division 

Corps 
planning 
center  of 
expertise 

Date  of 
peer 
review 
report(s) 

Feasibility 

or 

reevaluation 

study3 

Initiated 

after 

Nov. 

2005 

Estimated 

cost 

exceeding 
$45  million 

EIS 

Cost  of 
peer 
review 
contract 

Western  C-1 1 1 
Spreader  Canal 
Project 

Implementation 

Report 

Jacksonville 

South  Atlantic 

Ecosystem 

restoration 

Oct.  30, 
2009 

V 

V 

V 

184 

White  Oak  Bayou 
Flood  Damage 
Reduction  Plan 
General 
Reevaluation 

Report 

Galveston  - 
Local  sponsor 
led 

Southwestern 

Flood  risk 
management 

Jan.  11, 
2011 

V 

V 

124 

Wood  River  Levee 
System  Limited 
Reevaluation 

Report  for  Design 

Deficiency 

Corrections 

St.  Louis 

Mississippi 

Valley 

Flood  risk 
management 

Aug.  24, 
2011 

V 

V 

V 

79 

Source:  GAO  analysis  of  Corps  documents. 


Note:  In  addition  to  the  49  peer  reviews  included  on  this  list,  the  Louisiana  Coastal  Area  Beneficial 
Use  of  Dredged  Material  Program  Preliminary  Study  Report —  which  addressed  the  methodology  to 
select  and  prioritize  projects  under  the  program — underwent  peer  review  in  July  2008.  Section 
7006(d)  of  WRDA  2007  authorized  this  $100  million  program  for  the  beneficial  use  of  dredged 
material.  The  review  was  overseen  by  the  U.S.  Geological  Survey  and  conducted  by  the  Louisiana 
Coastal  Area  Science  Board,  and  about  $12,000  in  Corps  funds  were  used  to  support  its  completion. 
However,  because  the  completion  of  this  review  was  overseen  by  the  U.S.  Geological  Survey,  and 
did  not  involve  the  Corps’  typical  contract  process  or  costs,  it  was  not  included  in  our  list  or  analysis  of 
completed  peer  reviews. 

"Postauthorization  change  reports  are  generally  considered  reevaluation  reports,  and  project 
implementation  reports  are  generally  considered  feasibility  studies. 

bPeer  review  included  multiple  reports. 

cAs  part  of  this  contract,  a  follow  up  report  was  completed  in  July  201 0  to  determine  whether 
recommendations  from  the  peer  review  report  were  addressed.  This  follow  up  report  was  not  included 
in  our  analysis  because  it  was  not  provided  until  our  review  was  complete. 

dThe  eDNA  science  and  methodology  work  was  done  by  the  University  of  Notre  Dame,  and  the  study 
is  part  of  the  Great  Lakes  Regional  Initiative. 

ePeer  review  was  completed  under  multiple  contracts,  and  the  contract  cost  provided  is  the  sum  of 
the  cost  of  those  contracts. 

'Multiple  peer  reviews  were  conducted  under  one  contract  for  five  of  the  Louisiana  Coastal  Area 
studies. 

sThis  review  does  not  include  a  preliminary  report  done  by  Battelle  before  WRDA  2007.  Also, 
according  to  officials,  the  estimated  cost  of  the  project  for  this  study  is  not  applicable  (N/A)  because 
the  study  made  no  recommendations  for  construction. 
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hAs  part  of  this  contract,  a  report  on  Phase  I  and  Phase  II  of  the  project  were  completed.  The  Phase  II 
report  was  completed  in  April  2010  with  an  addendum  in  November  2010  but  was  not  included  in  our 
analysis  because  it  was  not  provided  until  our  review  was  complete. 
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DEPARTMENT  OF  THE  ARMY 

OFFICE  OF  THE  ASSISTANT  SECRETARY 
CIVIL  WORKS 
IDS  ARMY  PENTAGON 
WASHINGTON  DC  20310-0108 

reB  2  8  2012 


Ms.  Anu  K.  Mittal 

Director,  Natural  Resources  and  Environment 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 


Dear  Ms.  Mittal: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report, 
GAO-12-352,  “ARMY  CORPS  OF  ENGINEERS:  Peer  Review  Process  for  Civil  Works 
Project  Studies  Can  Be  Improved”  dated  February  9,  2012  (GAO  Code  361279).  Thank 
you  for  the  opportunity  to  review  and  comment  on  the  draft  report. 

We  concur  with  your  first  recommendation.  For  past  and  future  project  studies 
that  undergo  independent  external  peer  review  (IEPR),  the  Corps  will  list  the  specific 
statutory  authorities  under  which  the  peer  review  was  conducted  and  the  criteria 
triggering  the  IEPR  under  Corps  policy. 

We  partially  concur  with  your  second  recommendation  to  revise  the  criteria  in  the 
Corps’  process  for  conducting  peer  review  to  focus  on  larger,  more  complex  and 
controversial  projects,  to  encourage  peer  review  to  occur  earlier  in  the  study  process, 
and  to  include  exclusions  to  peer  review  that  align  with  section  2034.  The  Corps  is 
currently  in  the  process  of  reviewing  the  overall  efficiency  and  effectiveness  of  our  Civil 
Works  Review  Policy  (EC  1165-2-209).  The  findings  and  recommendations  of  this  GAO 
report  will  be  considered  along  with  numerous  other  factors  the  Corps  must  balance. 

We  concur  with  your  final  recommendation  to  develop  a  documented  process  to 
ensure  that  contractors  are  independent  and  free  from  conflicts  of  interest  on  a  project 
specific  basis. 

Responses  to  the  GAO  recommendations  are  enclosed. 


Enclosure 


Very  truly  yours, 


Jo-Ellen  Darcy 
Assistant  Secretary  of  the  Army 
(Civil  Works) 


Recycled  Paper 
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GAO  DRAFT  REPORT  -  DATED  FEBRUARY  9,  2012 
GAO  12-352  (GAO  CODE  361279) 


ARMY  CORPS  OF  ENGINEERS:  Peer  Review  Process  for  Civil  Works  Project  Studies 

Can  Be  Improved 

DEPARTMENT  OF  DEFENSE  COMMENTS  TO  THE  RECOMMENDATIONS 


We  generally  concur  with  the  GAO  recommendations  for  executive  action  (as  detailed 
below),  but  we  do  not  concur  with  all  of  the  GAO  findings  that  have  led  to  those 
recommendations.  Whereas  the  GAO  charge  is  focused  exclusively  on  section  2034  of 
WRDA  2007,  development  of  Corps  policy  must  be  responsive  to  all  of  our  applicable 
statutes,  administrative  directives,  and  professional  responsibilities.  We  believe  this  has 
resulted  in  a  stronger  policy  overall.  It  is  also  unfortunate  that  so  much  weight  in  the 
GAO  report  has  been  placed  on  anecdotal  remarks  from  field  level  officials  who  may  not 
have  had  the  benefit  of  the  corporate  vision  supporting  the  Army  Civil  Works  Program. 


RECOMMENDATION  1:  To  facilitate  Congressional  evaluation  of  the  7-year  trial  period 
outlined  in  section  2034,  the  Corps  should  identify  for  each  past  and  future  peer  review 
the  specific  statutory  authority  under  which  the  peer  review  was  conducted  and  the 
criteria  triggering  peer  review  under  the  Corps'  civil  works  review  policy. 

POD  RESPONSE:  CONCUR.  The  Corps  Civil  Works  Review  Policy  (EC  1 165-2-209) 
establishes  an  accountable,  comprehensive,  life-cycle  review  strategy  for  Civil  Works 
products  by  providing  a  seamless  process  for  review  of  all  Civil  Works  projects  from 
initial  planning  through  design,  construction,  and  operation,  maintenance,  repair, 
replacement  and  rehabilitation.  As  such,  the  policy  must  satisfy  not  only  the  Chief  of 
Engineer's  inherent  responsibility  for  ensuring  project  safety  and  quality  of  the  products 
USACE  provides  to  the  American  people,  but  also  the  numerous  statutes  applicable  to 
Corps  Civil  Works  activities.  The  Civil  Works  Review  Policy  addresses  the  Office  of 
Management  and  Budget  (OMB)  peer  review  requirements  under  the  "Information 
Quality  Act"  and  the  Final  Information  Quality  Bulletin  for  Peer  Review.  It  also  provides 
guidance  for  the  implementation  of  both  Sections  2034  and  2035  of  the  Water 
Resources  Development  Act  (WRDA)  of  2007  (Public  Law  (P.L.)  1 1 0-1 14).  For  past 
and  future  project  studies  that  undergo  independent  external  peer  review,  the  Corps  will 
develop  and  implement  an  approach  to  list  the  specific  statutory  authorities  under  which 
the  peer  review  was  conducted  and  the  criteria  triggering  the  IEPR  under  Corps  policy. 


RECOMMENDATION  2:  To  better  reflect  section  2034  and  provide  more  effective 
stewardship  of  public  resources  and  ensure  efficient  and  effective  operations,  the  Corps 
should  revise  the  criteria  in  the  Corps  process  for  conducting  peer  review  to  focus  on 
larger,  more  complex  and  controversial  projects,  to  encourage  peer  review  to  occur 
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earlier  in  the  study  process,  and  to  include  exclusions  to  peer  review  that  align  with 
section  2034. 

POD  RESPONSE:  PARTIALLY  CONCUR.  The  Corps  is  currently  in  the  process  of 
reviewing  the  overall  efficiency  and  effectiveness  of  our  Civil  Works  Review  Policy.  The 
findings  and  recommendations  of  this  GAO  report  will  be  considered  along  with  those  of 
several  other  ongoing  internal  Corps  reviews  which  are  directed  at  the  full  range  of 
product  review,  not  just  independent  external  peer  review  (IEPR)  for  project  studies. 

In  general  we  agree  that  IEPR  should  be  applied  to  studies  that  will  significantly  benefit 
from  independent  external  peer  review.  Now  that  we  have  a  track  record  to  appraise, 
we  will  reassess  our  criteria  and  how  the  criteria  are  applied  in  determining  which 
studies  should  undergo  IEPR. 

Regarding  timing  of  reviews,  the  Corps  agrees  that  initiating  reviews  early  in  the 
process  is  advantageous.  However,  early  involvement  of  review  panels  must  be 
balanced  with  having  sufficient  data  and  analysis  available  for  review.  The  iterative 
planning  process  also  complicates  the  scheduling  and  contracting  process  for  IEPR 
panel  members.  Our  existing  Civil  Works  Review  Policy  states  that  “To  the  maximum 
extent  practicable,  reviews  shall  be  scheduled  and  conducted  early  in  the  process  .... 
This  is  particularly  pertinent  in  the  case  of  independent  external  peer  reviews."  We  note 
that  in  implementing  external  peer  review  policy  in  2005,  many  studies  that  were 
already  well-along  in  their  process  now  faced  requirements  to  conduct  IEPR.  This 
resulted  in  a  “backlog”  of  sorts  which  we  believe  accounts  for  the  fact  that  a 
disproportionate  number  of  studies  in  this  audit  had  initiated  IEPR  relatively  late  in  the 
study  process.  The  Corps  is  currently  undertaking  a  major  overhaul  of  its  planning 
processes  (Civil  Works  Transformation)  which  is  working,  among  other  things,  to 
shorten  the  planning  study  process  and  to  better  align  product  reviews  for  greater 
overall  effectiveness. 

Regarding  exclusions  to  peer  review,  the  Corps  does  not  concur  with  the  GAO  findings 
that  the  Corps  process  does  not  include  the  flexibility  provided  in  section  2034,  and  that 
some  studies  that  have  undergone  peer  review  did  not  warrant  it.  The  three  instances 
specifically  cited  by  GAO  (Green  Bay  Dredge  Material  Management  Plan,  Wl;  Chacon 
Creek  Flood,  TX;  and  Yuba  River  General  Re-evaluation  Report,  CA)  were  each 
carefully  deliberated  in  support  of  the  agency  decision,  and  the  Corps  stands  by  its 
decisions  to  conduct  IEPR  on  these  studies.  We  also  stand  by  all  of  the  decisions  we 
made  for  granting  or  denying  IEPR  exclusion  to  date.  Nonetheless,  as  part  of  our 
overall  review  of  the  Civil  Works  Review  Policy,  we  will  assess  the  effectiveness  of  our 
criteria  and  how  the  criteria  are  applied  toward  determining  which  studies  should  be 
considered  for  exclusion. 
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Finally,  in  addition  to  the  considerations  above,  refinements  to  the  Civil  Works  Review 
Policy  must  be  developed  to  reflect  the  emphases  within  other  sections  of  WRDA  2007 
as  well  as  all  other  pertinent  statutes  and  Administration  priorities.  Within  WRDA  2007, 
for  example,  sections  2031,  2032,  2033  and  2035  all  emphasize  public  safety,  while 
section  2033  establishes  a  benchmark  goal  for  significantly  shorter  study  timeframes. 

In  balancing  all  of  these  requirements  and  strategic  directives,  the  Corps  will  continue  to 
improve  its  review  processes  to  provide  procedures  for  ensuring  the  quality  and 
credibility  of  U.S.  Army  Corps  of  Engineers  decision,  implementation,  and  operations 
and  maintenance  documents  and  work  products. 


RECOMMENDATION  3:  To  better  reflect  section  2034  and  provide  more  effective 
stewardship  of  public  resources  and  ensure  efficient  and  effective  operations,  the  Corps 
should  develop  a  documented  process  to  ensure  that  contractors  are  independent  and 
free  from  conflicts  of  interest  on  a  project  specific  basis. 


POD  RESPONSE:  CONCUR.  The  Corps  will  develop  and  implement  a  more 
transparent  and  better  documented  process  for  ensuring  that  contractors  are 
independent  and  free  from  conflicts  of  interest  on  a  project  specific  basis.  Our  current 
process  already  has  many  safeguards  in  place,  but  we  will  institute  an  additional 
process  for  obtaining  information  from  contractors  on  a  project  specific  basis,  and  for 
reporting  that  information  in  project  documentation.  We  will  base  the  approach  on  the 
protocols  that  OMB  and  the  National  Academies  of  Science  have  established  for 
addressing  independence  and  conflict  of  interest  for  individual  panelists. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (www.gao.gov).  Each  weekday  afternoon, 
GAO  posts  on  its  website  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 
Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 
7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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